Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis

Abstract: We introduce DMTet, a deep 3D conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such as coarse voxels. It marries the merits of implicit and explicit 3D representations by leveraging a novel hybrid 3D representation. Compared to the current implicit approaches, which are trained to regress the signed distance values, DMTet directly optimizes for the reconstructed surface, which enables us to synthesize finer geometric details with fewer artifacts. Unlike deep 3D generative models that directly generate explicit representations such as meshes, our model can synthesize shapes with arbitrary topology. The core of DMTet includes a deformable tetrahedral grid that encodes a discretized signed distance function and a differentiable marching tetrahedra layer that converts the implicit signed distance representation to the explicit surface mesh representation. This combination allows joint optimization of the surface geometry and topology as well as generation of the hierarchy of subdivisions using reconstruction and adversarial losses defined explicitly on the surface mesh. Our approach significantly outperforms existing work on conditional shape synthesis from coarse voxel inputs, trained on a dataset of complex 3D animal shapes. Project page: https://nv-tlabs.github.io/DMTet/.


Reviews and author responses:

 I think we look at the issues from different angles. I look at these issues from the perspectives of guarantees or at least theoretical insights, while the authors argue from the empirical perspectives. Ultimately, it is the call of area chairs.

For example, I can understand that for simple shapes, self intersection or topology is not a big issue. However, it becomes important for complex shapes with thin structures, e.g., Engines and plants. Also, there is issue in out-of-sample generation.

 

 We thank reviewer YB43 for agreeing with our reply. Also, we appreciate reviewer f9sZ for summarizing the main concerns. Unfortunately, despite we have clarified each concern in the previous reply followed by more clarification from reviewer YB43, reviewer f9sZ does not show agreement or disagreement with our arguments, and insists on the same arguments which we already addressed. Thus we reinforce our arguments to each point here:
1. We agree with the statement, but we don’t see what the concern is. We provide comprehensive and fair comparisons with existing 3D representations in our paper (Tab. 1 and 3) while using SOTA network architecture to encode implicit fields in both baselines and DMTET. 

    In addition, the reviewer seems to imply that the discussion on “underlying principle” is missing. However, we think our contribution is well motivated and analyzed. While existing works on differentiable iso-surfacing [26, 38] have shown that directly supervising the underlying surface of implicit representation leads to better results (e.g. finer details, closer aligned surface), their formulation does not scale up to high-resolution in a training setup. We propose using MT as an alternative and analyze its differentiability in Section A. In contrast to the analysis in [26,38], we show MT allows effective topological changes. Combining it with selective subdivision and deformable tetrahedral grid, MT can scale up to high-resolution. We apply our representation to 3D shape synthesis tasks and show that supervision on the extracted surface improves the shape synthesis quality. This confirms the prior works and we compare with [26, 38]. We kindly ask the reviewer to point out which specific contribution above is considered not justified. 

2. We clearly explained and demonstrated that DMTET can **mitigate** the **topological error** introduced by current iso-surfacing methods for **implicit approach**, allowing it to represent topological features smaller than grid resolution. Here we refer to our previous reply:
> The reviewer agrees that the topological error from discretization (MC/MT) is inevitable for the implicit approach when a surface mesh is desired as output (to, say, use the result in a graphics pipeline). The supervision in implicit space is not aware of the discretization error introduced by the iso-surfacing step, thus cannot address the issue. Optimizing directly for the reconstructed surface via differentiable MT layer can mitigate this error as we illustrate in 2D: https://dmtet.s3.us-west-1.amazonaws.com/topological_err.mp4. We do not need explicit supervision on topology as suggested by the reviewer.

    and reply from reviewer YB43:
> With regard to SIREN, maybe the concern is that a purely implicit neural representation could already be sufficient? In such a case, it is my understanding that subsequent surface extraction is an issue as Sec. 4.2.1 argues by assuming an oracle, e.g. a perfectly trained SIREN I suppose. DMT however makes the implicit representation aware of the surface extraction step, which even a carefully designed implicit representation based on SIREN that is trained without any information about extraction won't be.

    Moreover, we wish to clarify that **self-intersection** is not a limitation of our hybrid representation. It is produced by surface subdivision module in rare cases, but the module is **optional** for modeling parametric surfaces. The core of our representation, differentiable MT layer, does not produce self-intersecting meshes if the tetrahedral grid does not have intersecting volume. In our chosen grid pattern, this is guaranteed if the magnitude of the deformation vector is hard constrained to be smaller than $\frac{1}{2}$ of the edge length of tetrahedra. We benchmark the performance without surface subdivision/grid deformation in Tab. 3, which demonstrates the improvement from joint learning.

    In summary, we argue that our hybrid representation is more performant than implicit approach when an explicit surface is desired as output, and we do not consider it introducing additional limitations to the existing implicit pipeline (dense sampling + iso-surfacing via MC/MT). If the reviewer finds which part of our explanation and rebuttal is not clear, please let us know.

3. We do not claim better generalization in our paper, thus we are confused by what is required by the reviewer. We refer to the argument from  reviewer YB43:
> That fine details can disappear is indeed a downside. But it is also a downside shared with existing methods for this task, and I currently don't have reason to believe that DMT is worse in this regard.

    We thank the reviewer for suggesting additional regularization losses, but we don’t think this issue hurts the contribution of our work. Moreover, all quantitative and qualitative evaluations are conducted on unseen shapes during training, not training instances as the reviewer seems to imply.


 I would like to narrow down a few comments here. 

1. 3D representation, in the deep learning era in particular, is not correlated to the task. Basically, 3D representations encode 3D data into vectorized forms, among which we apply machine learning techniques. I think whenever a representation allows dense predictions, it can be modified to the task of 3D generative modeling. 

A tradition in vision is to evaluate different approaches on benchmark datasets. There it cares more about the efforts that adopt ml techniques to different tasks. This is why I recommended the authors look into those. In contrast, the tradition of ML conferences is about underlying principles.

2.  From my understanding, the output of the proposed approach cannot guarantee self-intersection free. Moreover, all topological features whose size that are smaller than the grid resolution cannot be modeled. Moreover, the implicit representation can initialize a shape with a very bad topology, which the mesh representation cannot be fixed (more on this below). There are fundamental limitations. I am negative because usually, when using multiple representations, one should combine the strength of each representation. The proposed approach loses the strength of each representation in the end.

3. The argument from the author is that the best strategy is to use a fine grid for the implicit representation, convert it into a mesh representation, and enforce the training loss on the mesh representation. This strategy will have better generalization than alternative strategies. This is indeed interesting. However, I cannot understand why this is the case. So simple stability analysis would be necessary for an ML conference. 

     Alternatively, I proposed that one has to enforce some regularization losses on synthetic instances to address this issue. Note that it is not vital that the topologies of training instances are correctly reconstructed, but rather the topologies of synthetic instances should be reconstructed. The animation does not show the generalization ability. 



 I agree with the authors' response to the points raised in the main review. It is a downside of purely implicit representations that they don't allow to directly work with the explicit surface in the loss function since the surface is not accessible without first finding it (via dense sampling, iterative root finding, or maybe other means). I also prefer the harder coupling that DMT enables over the soft coupling of "Coupling Explicit and Implicit Surface Representations for Generative 3D Modeling. ECCV 2020." That work's main argument against Deep Marching Cubes is its cubic memory requirements, an issue that DMT addresses via selective subdivision. I'm more aware of "hybrid" as referring to combining multiple representations (especially implicit and explicit) in some way, not necessarily just their potentially rather independent predictions in an ensemble sense to leverage complementary advantages. E.g. the following work does not take the ensemble route: "Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations. CVPR 2021.", which could also be cited by this submission.

In the reviewer's response, I unfortunately have to agree with the authors that it is unclear to me what the concerns are.

With regard to SIREN, maybe the concern is that a purely implicit neural representation could already be sufficient? In such a case, it is my understanding that subsequent surface extraction is an issue as Sec. 4.2.1 argues by assuming an oracle, e.g. a perfectly trained SIREN I suppose. DMT however makes the implicit representation aware of the surface extraction step, which even a carefully designed implicit representation based on SIREN that is trained without any information about extraction won't be.

In the context that this submission works in, there is not a sense in which there are geometric features that are not captured by the input grid, unless I'm misunderstanding something about the work. The input grid is the input. Only for the purpose of *evaluating* the method, the input grid is generated from detailed shapes. But the actual test time usage assumes an artist sculpting a nice coarse grid, which is enough to get a detailed result as the submission shows. Does the concern maybe relate to the artist's desiring certain details that aren't representable by the input? Subsequent refinement by the artist is an interesting aspect that future work could tackle similar to e.g. "Interactive 3D Modeling with a Generative Adversarial Network", which could also be cited in the limitations, as a potential way of making the result editable.

Unfortunately, the only other point that I am able to follow is point 4. That fine details can disappear is indeed a downside. But it is also a downside shared with existing methods for this task, and I currently don't have reason to believe that DMT is worse in this regard.

 We are somewhat confused about what the reviewer is finding problematic. 
> The paper does not convince me that combining the implicit representation and the explicit representation in the proposed approach is superior to using a single implicit representation.

We agree with the reviewer that “one can model various shape details under the implicit representation”, and we wish to clarify that we do **not** claim that combining implicit representation with explicit representation improves the **capability** in representing surface details. This is not our point at all. Our main focus is on learning-based 3D shape **generation** problems, where the network needs to learn how to generate plausible 3D shapes conditioned on coarse voxels or point clouds. This is fundamentally different from representing/overfitting one specific shape, where we do not need to generalize to predict novel shapes at test time.  Each GT shape in our target dataset can be overfitted almost perfectly with our baseline implicit model, so the capacity of representation is not the problem here.  In our case, our main contribution is in differentiably converting implicit representations into explicit surface meshes, which allows us to incorporate supervision on the surface during training -- this helps the network to better learn to generate shape details (see below). We empirically show this in Table 1 and 3.

The reviewer acknowledges that
> In general, for 3D deep learning, an equally important issue as representations is how to develop effective losses to train the deep model. 

We are perfectly in agreement with the reviewer, and this exactly describes our goal and main contribution! We utilize a differentiable iso-surfacing layer (MT) to enable losses to be defined on the surface parametrized by the implicit field. Our motivation follows [26, 38] that show that directly supervising the underlying surface leads to better results (e.g. finer details, closer aligned surface) than solely regressing SDF/Occupancy values, **in learning-based generative tasks**. This is enabled by our MT layer, but better than the previous alternatives [26,38]. We also demonstrate this in our paper (see table 1, 2 and 3) where we utilize an additional chamfer loss or adversarial loss defined on the explicit surface -- we achieve much better performance than baselines both quantitatively and qualitatively. Note that here we compare against the state-of-the-art implicit approach in a learning-based application while the surface is extracted at high-resolution. 

We do strongly feel that we adequately and fairly benchmark our method against existing alternatives on the target applications. We again stress that the suggested SIREN and MPU are not the right baseline here, since they do not show improvement to large-scale 3D reconstruction/generation tasks. If the reviewer has a more appropriate baseline to suggest that fits our target application, we would be more than happy to include it. 


Here we address the other concerned comments: 

[Topological Error] The reviewer agrees that the topological error from discretization (MC/MT) is inevitable for the **implicit approach** when a surface mesh is desired as output (to, say, use the result in a graphics pipeline).  The supervision in implicit space is not aware of the discretization error introduced by the iso-surfacing step, thus cannot address the issue. Optimizing directly for the reconstructed surface via differentiable MT layer can mitigate this error as we illustrate in 2D: https://dmtet.s3.us-west-1.amazonaws.com/topological_err.mp4. We do *not need* explicit supervision on topology as suggested by the reviewer.

[Inconsistency in the text] We do **not** claim that we use surface subdivision to add local details. Instead, we want to model the parametric surface with fewer control points for improving efficiency during inference. For example, extracting a smooth sphere from a deep implicit model requires densely evaluating SDF near the iso-surface. In contrast, one can extract a simple icosahedron from SDF and reconstruct the sphere with subdivision whose computation cost is insignificant compared to inferring SDF values with deep implicit networks. This is one advantage of having an explicit surface, which is acknowledged by the reviewer --  *“if the output requires a surface parameterization, then this has to be done under the explicit representation”*.

[Input initial grid] We sometimes observe reconstructed thin structures being broken, which is also a limitation of the current generative approach. Adding the proposed surface supervision can alleviate this issue (see Fig. F last row and Fig. 7 second row comparing ConvOnet’s result and ours), but not fully address it. We would like to explore other regularization terms in future work as suggested by the reviewer.

[Eqn. 6] is not a smoothness loss as argued by the reviewer. Notice that $\Delta s(v_i)$ is the predicted vertex deformation to the tetrahedral grid and surface mesh. Since running on the initial grid would not produce self-intersecting mesh, limiting the deviation from it can alleviate artifacts.


 [ we are tackling the problem of generating high-quality shapes from low-resolution voxels or point clouds] I was referring to the representation used in prior works such as SIREN. As I will describe below, it is unclear what is the advantage of adding the mesh representation in the way that is proposed in this paper.

[Error propagation from implicit space to mesh space] If we use a coarse grid, then the topologically features who 'size' is below the grid resolution cannot be captured by the approach even there is a very fine-grained mesh. If we use a fine grid, then the topological error introduced by the fine grid cannot be addressed by the mesh representation as well. I have some suggestions on this, but I cannot understand how joint learning can address this issue. 

Also, eq (6) is a smoothness loss which does not address the self-intersection issue (different from flipping triangles). 

[Comparisons to related works] This is a minor issue. But when you are talking about hybrid representations, it would be great to discuss all relevant works.

[Input initial grid] Thanks for submitting the gif. It is clear that when perturbing the initial grid, the topology and the geometry of the output can change in undesired ways. See my comments below. 

[Inconsistency in the text] It is unclear to me why adding local details cannot be accomplished using a carefully designed implicit representation. 

Suggestions:

1. The paper does not convince me that combing the implicit representation and the explicit representation in the proposed approach is superior to using a single implicit representation. For example, one approach is to augment the SDF representation with additional information such as normal/curvature directions encoded at the grid nodes and using those to model shape details. The SIGGRAPH 2003 paper  Multi-level Partition of Unity Implicits has shown that one can model various shape details under the implicit representation. One can imagine that neural implicits can achieve similar or even better results.

2. To really address the topology issue, we should add another regularization loss that aligns the distribution of the topologies of the synthetic shapes and that of the training data. This can provide efficient regularizations when the genus of the training data is small, while the synthetic data has a lot of holes (without regularization). 

I am just giving one suggestion here. In general, for 3D deep learning, an equally important issue as representations is how to develop effective losses to train the deep model.

3. The explicit representation can be complementary to the implicit representation in many ways. For example, if the shape has primitive structures, then it is easier to enforce those structures under the explicit representation than under the implicit representation. As another example, if the output requires a surface parameterization, then this has to be done under the explicit representation. 

4.  The animation shows that this conditional refinement task is rather difficult. The resulting mapping from the coarse grid to the final mesh is not that smooth, e.g., one of the legs of the cow model is broken. It would be good to think about how to add some regularization term to avoid this issue. 

Overall, I still do not think this paper is ready for publication at this point. Given that the underlying machine learning principle of the proposed approach is unclear, I would recommend submitting it to a vision conference. I am happy to hear additional thoughts from the authors.

 We thank the reviewer for the positive and detailed comments. Please find our response to each comment below:

[Technical contribution] We apologize for the confusion of the contribution and summarize it below. We will make it clear in the revision.
1)  In contrast to the analysis in (DMC [26], MeshSDF[38]), we show that using marching tetrahedra (MT) as a differentiable iso-surfacing layer allows topological change, and compare with [26, 38]. To our best knowledge, we are the first to introduce MT as a differentiable module in a DL framework, and demonstrate that the additional supervision (e.g. chamfer distance, adversarial loss) defined directly on the extracted surface improves the shape synthesis quality. 
2) The original MT (or MC) algorithm has a cubically growing computation cost and thus does not scale up to high-resolution in a training setup. We introduce a coarse-to-fine optimization strategy that achieves better reconstruction quality than state-of-the-art methods, while requiring a lower computation cost. Our method is composed of modules that extend existing work to allow end-to-end training and address the known limitations (discussed in 3.2.1).

[Training Time] Our model is optimized with main modules implemented in CUDA, so it achieves fast runtime at inference. The major bottleneck during training is the computation of distance-based metrics, such as Chamfer Distance and Signed Distance, which needs to compute point-to-point and point-to-face distance for millions of pairs. Currently in our implementation we compute the distance on the fly. It is possible to speed up training by using e.g. pre-computed SDF values or sampling fewer points when computing CD (currently we sample 100k points).

[Gradient outside current Tetrahedra] We thank the reviewer for pointing this out. We agree that our formulation of MT will not have gradients outside current tetrahedra, however during training, we observe the network can gradually deform the surface and approach the target shape (see the visualization in Fig. A).  We further provide a 2D example here https://dmtet.s3.us-west-1.amazonaws.com/bunny.mp4 where we optimize Chamfer Distance without L_sdf (GT points are in purple and the predicted surface is in red). 

[Evaluation Dataset] We evaluated our model on both animals (Sec. 4.1) and ShapeNet (Sec. 4.2) dataset.
We additionally evaluate our voxel upsampling method on ShapeNet in the rebuttal, and compare it with Conv OccNet. We report L1 Chamfer Distance in the table below, and will include experimental details in the revision.

\begin{array}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}  
\displaylines{ \hline Category & Airplane & Bench & Dresser & Car & Chair & Display &Lamp & Speaker & Rifle & Sofa & Table & Phone & Vessel & Mean\\\ \hline  ConvOnet[15] &1.63&2.06&2.19&1.88&2.10&2.06&2.59&2.31&4.74&2.11&2.12&1.73&2.17&2.28 \\\ \hline DMTET &\textbf{1.30}&\textbf{1.70}&\textbf{2.01}&\textbf{1.72}&\textbf{1.93}&\textbf{1.96}&\textbf{1.89}&\textbf{2.18}&\textbf{4.43}&\textbf{1.96}&\textbf{1.97}&\textbf{1.68}&\textbf{1.88}&\textbf{2.04} \\\ \hline }
\end{array}

[Minor clarifications] We thank the reviewer for pointing out these problems, we address each of them (in the same order) below.  We will fix these in the revision.

·        f(v_i) is the updated per-vertex feature that overwrites the previous f(v_i) which serves as input to GCN. We will denote the updated feature with another symbol.

·        Volume Subdivision is applied once in all experiments. We apply surface subdivision once during training and compute the loss on the resulting surface. We apply surface subdivision twice at test time. We found this to be sufficient to capture the details in tasks we are tackling.

·        GCN for Learnable Surface Subdivision does not take f(v_i). We apply another update with GCN after the volume subdivision step, which takes updated f(v_i) as input. The f(v_i) of the new vertices from volume subdivision is computed by averaging the SDF values on the edge of the tetrahedron they lie on.

·        Yes, it is the same as in [10]. We will include them in the revision.

·        v'_i and alpha_i are only predicted once by GCN in the beginning. The idea is to control the subdivided surface with predictions only on control points (vertices before subdivision) for Loop Subdivision to maintain low computation cost.

·        Yes, higher alpha leads to smoother surfaces. A vertex with alpha = 0 will not change its initial position in the subdivision steps, thus our learnable subdivision module is able to preserve shape features. We will include a comparison with the original Loop Subdivision in a revision.

·        The small noise is added to the position of v s.t. v’ = v + noise $\in R^3$ and we extract a local SDF grid around v’. Specifically, v appears in equations (4) and (5) should be replaced by v’. The purpose is to shift the sampled SDF grid to avoid SDF(v, M_gt) always equals 0, which makes it trivial for D to classify it as a real sample.

·        We compute the point-triangle distance to all triangles in the mesh and take the smallest distance. The final SDF value is this distance times the occupancy of the point.

·        We will move it to the main paper to avoid confusion.

·        We will fix this in our revision.



 We thank the reviewer for the comments. Please find our response to each comment below:

[Technical contribution] While we do exploit the idea of deforming a tetrahedral grid from DefTet [16], we do not claim this as a contribution. Our technical contributions include:
1)  In contrast to the analysis in DMC [26] and MeshSDF[38], we show that using marching tetrahedra (MT) as a differentiable iso-surfacing layer allows topological change, and compare with [26, 38]. To our best knowledge, we are the first to introduce MT as a differentiable module in a DL framework, and demonstrate that the additional supervision (e.g. chamfer distance, adversarial loss) defined directly on the extracted surface significantly improves the shape synthesis quality. 
2) The original MT (or MC) algorithm has a cubically growing computation cost and thus does not scale up to high-resolution in a training setup. We introduce a coarse-to-fine strategy that achieves better reconstruction quality than state-of-the-art methods, while requiring a lower computation cost. 
 
[Typological change] We extensively demonstrate that the gradient signal from MT can change the typology of extracted mesh during training/optimization.
 1) Fig. A in the supplementary material shows that the surface mesh topology changes smoothly when optimizing only chamfer distance. We provide additional visualization in 2D (optimizing Chamfer Distance) to illustrate the process: https://dmtet.s3.us-west-1.amazonaws.com/bunny.mp4 (GT points are in purple and the predicted surface is in red)
2) In ablation study, we compare the reconstructed shape trained with Occupancy/SDF loss with the prediction of the model trained with additional supervision on the surface extracted by MT (Fig. 7 and Fig. E) 
 3) Our GAN loss is propagated from the predicted mesh to the network weights via our MT layer, which results in topological change to better model the details. (e.g. second row Fig. F)

[Evaluation Dataset] While we evaluate our voxel upscaling method on shapes from Turbosquid, we also evaluate the proposed method on the Point Cloud Reconstruction task on ShapeNet and compare it with a variety of existing approaches (Sec. 4.2). We demonstrate better reconstruction quality with our proposed method. As suggested by the reviewer, we additionally evaluate our voxel upsampling method on ShapeNet in the rebuttal, and compare it with ConvONet[15]. We report L1 Chamfer Distance in the table below, and will include experimental details in the revision.

\begin{array}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}  
\displaylines{ \hline Category & Airplane & Bench & Dresser & Car & Chair & Display &Lamp & Speaker & Rifle & Sofa & Table & Phone & Vessel & Mean\\\ \hline  ConvOnet[15] &1.63&2.06&2.19&1.88&2.10&2.06&2.59&2.31&4.74&2.11&2.12&1.73&2.17&2.28 \\\ \hline DMTET &\textbf{1.30}&\textbf{1.70}&\textbf{2.01}&\textbf{1.72}&\textbf{1.93}&\textbf{1.96}&\textbf{1.89}&\textbf{2.18}&\textbf{4.43}&\textbf{1.96}&\textbf{1.97}&\textbf{1.68}&\textbf{1.88}&\textbf{2.04} \\\ \hline }
\end{array}

 We thank the reviewer for the detailed and positive comments. We address the questions below: 

[Generalization to Out-of-Distribution Data] – We downloaded a set of 20 artist-created shapes from the TurboSquid dataset (available here: https://www.turbosquid.com/3d-models/3d-model-voxel-animals-1301280), and tested on all of them. We observe good performance on the mammal shapes that fall into our trained animal categories (e.g. bear, giraffe), which we are showing in the paper. For the non-mammal animals that are very different from our training distribution (e.g. frog and crocodile), our predictions are not perfect, but still reasonable. We include visualization for all examples here: https://dmtet.s3.us-west-1.amazonaws.com/animal.mp4

 [Related Works] We thank the reviewer for pointing out several references. We will add the following discussion in our revision:

"Neural Subdivision" [Liu et al., SIGGRAPH 2020] - There are two major differences between our surface subdivision and neural subdivision: 1) neural subdivision is trained on a pre-computed 1-1 vertex mapping, whereas ours can be trained with any objective functions defined on a surface e.g. $L_{CD}$, GAN loss. 2) Neural subdivision requires running inference on the new mesh generated in each subdivision step, while our learnable surface subdivision module only predicts per-vertex parameters at the beginning, which controls the iterative subdivision process, and runs Loop subdivision afterwards. Thus, our module can attain a lower computational cost.

"Neural Unsigned Distance Fields for Implicit Function Learning" [Chibane et al., NeurIPS 2020] - This paper mainly focuses on using an unsigned distance field (an implicit approach) to representing the shape, while we try to use MT to bridge the gap between implicit and explicit representations, enabling supervision defined on the explicit surfaces.

"SAL" [Atzmon et al., CVPR 2020] does not require explicit supervision on the occupancy, however, the method requires a careful initialization of network parameters, making it hard to scale up to complex dataset, where our method is directly trained from scratch.

"Analytic Marching" [Lei et al., ICML 2020] - Analytic Marching is trained with SDF loss, whereas our representation can be supervised by loss defined on the extracted surface.

[Typos]: Thanks for the suggestions, we will address them in a revision.


 We thank the reviewer for the feedback. We address the comments below. There are several misunderstandings that the reviewer has regarding our paper, that we wish to further clarify.

First, we emphasize that we are tackling the problem of **generating** high-quality shapes from low-resolution voxels or point clouds, instead of approximating one particular shape by overfitting a neural network as the reviewer seems to imply. Thus SIREN (Implicit Neural Representations with Periodic Activation Functions. NeurIPS 2020), which only showed state-of-the-art performance on overfitting individual shapes, is not applicable to our problem. We do compare with relevant SOTA methods (ConvOccNet[37], DefTet[16]) in our experiments (Tab. 1 and Tab. 3).

[Error propagation from implicit space to mesh space] We kindly disagree with this critique. We want to clarify that our hybrid representation is not a two-stage pipeline as the reviewer seems to imply. Our key contribution is in utilizing MT as a differentiable layer that converts an implicit representation into an explicit mesh on the fly. This allows training the implicit representation with loss defined on the surface (e.g. Chamfer loss and adversarial loss), which learns better features and captures geometric details.  

The reviewer argues that our representation is limited due to the use of a coarse grid to discretize implicit field. However, our representation (MT layer, in particular) is applicable at any grid resolution. In addition, joint optimization can alleviate the topological error from discretization (see Fig. 8 and Fig. 9), which is inevitable when reconstructing the shape from implicit representation. We found the current grid resolution sufficient to represent accurate topology in both tasks.
We thank the reviewer for pointing out the limitation of self-intersection. We introduce one regularization to avoid this (L_{def} in Eq. 6), and leave the guarantee of no self-intersection for future work. 

[Comparisons to related works] Thanks for pointing out the related works. In contrast to Poursaeed et al. [Coupling Explicit and Implicit Surface Representations for Generative 3D Modeling. ECCV 2020], the loss on the surface (e.g. Chamfer Distance, adversarial loss, and image loss) is directly propagated to SDF instead of via the consistency loss, which we argue is more effective since 1) we do not need to train a separate network to render the mesh, and 2) The learning signal corresponds to the surface represented by an implicit function, not the intermediate one produced by a mesh branch as in Poursaeed et al.

[Input initial grid] In the shape synthesis from voxels task, the input to the network is a point cloud sampled from the surface of a coarse voxel grid (we use 16x16x16 resolution during training). Therefore, our model can take in a coarse shape with different resolution than the training resolution, as demonstrated in Fig. 6. All shapes in Fig. 6 are created by an artist and have noticeable differences with our coarse voxels used in training, e.g., different ratios of body parts compared with our training shapes (larger head, thinner legs, longer necks). Our model faithfully generates high-quality 3D details conditioned on each coarse voxel shape. Following the suggestion from the reviewer, we include a robustness test by running inference on a sequence of input shapes, each modified by a few blocks at a time: https://dmtet.s3.us-west-1.amazonaws.com/voxel_interp.gif. Our model generates plausible shape after each small perturbation, while respecting the human input (e.g., broken leg).

[Inconsistency in the text] We apologize for the confusion and clarify it here: Our learnable subdivision module is proposed to address the two limitations in loop subdivision scheme: 1) inability of preserving sharp features, and 2) volume shrinkage. In the loop scheme, a parameter $\alpha$ is computed for each vertex which is always larger than 0. Then, the vertex position is updated by averaging neighboring vertices as in Fig. B, which smooths out sharp features in the mesh. In our proposed module, this parameter is learnable and predicted by a network, thus sharp features can be preserved by predicting $\alpha$ close to zero. In particular, we use a graph-neural-network to predict the value of $\alpha$, and run the Loop Subdivision using the predicted  $\alpha$ to get a parametric surface. This module is differentiable, so loss defined on subdivided surface can be propagated back to the implicit field through MT to optimize the topology.


The paper presents a method for generating high-resolution 3D shapes guided by coarse voxel-based input. The key idea is to combine tetrahedral meshes which are adaptive and more efficient, with sign-distance field, along with a differentiable marching tetrahedra layer that bridges the gap between tetrahedra and surface meshes. The method achieves improved performance compared with existing methods.   The paper presents a method for generating high-resolution 3D shapes guided by coarse voxel-based input. The key idea is to combine tetrahedral meshes which are adaptive and more efficient, with sign-distance field, along with a differentiable marching tetrahedra layer that bridges the gap between tetrahedra and surface meshes. Overall, the method is plausible and shows decent performance. I have the following concerns:

From a technical perspective, the method is built on DefTet [16], by introducing SDF and Marching Tetrahedra. All these components are from existing publications, which somehow affects the novelty of the method. Nevertheless, the approach to combine these leads to improved performance and is itself interesting.

Although the paper claims that the method can handle topological changes, I can't seem to find examples that clearly demonstrate this. Moreover, a large part of the assessment is based on a dataset collected by authors, which makes comparisons more challenging. Evaluation of guided synthesis on more public datasets would make it more convincing. 
 N/A

This paper introduces a shape refinement approach that first uses an implicit representation to obtain an initial shape. The initial shape is then converted into a mesh representation refined by loop subdivision.   The paper basically combines a couple of hand-crafted designs that are not clearly motivated. Although the experimental results are ok (not outstanding, see my comments below), the underlying principles from the machine learning and statistical aspects are rather unclear. Below are the detailed comments:

First of all, a benefit of the implicit representation is that it is known to capture shape details. This paper uses it as initialization. The subdivision scheme produces smooth outputs that are not feature preserving under the loop subdivision scheme. 

It would be good to compare the proposed approach that state-of-the-art neural implicit representations for shape synthesis. For example, the comparisons to the following paper is necessary.
Implicit Neural Representations with Periodic Activation Functions. NeurIPS 2020. 

The notion of hybrid is different from many other works that combine predictions under different representations. In this case, the errors made under one representation can be rectified under other representations or different representations help learn better features under a multi-task learning framework. Such approaches are grounded by bias and variance of optimizing over-parameterized networks, where ensembles can help reduce the variance:
H3dnet: 3D Object Detection using Hybrid Geometric Primitives. ECCV 2020.
Coupling Explicit and Implicit Surface Representations for Generative 3D Modeling. ECCV 2020.
In contrast, the proposed approach is sequential, meaning that the topology mistakes of the implicit representation cannot be rectified in the mesh representation. On the other hand, the mesh representation cannot guarantee self-intersection free during subdivision, which is an advantage of the implicit representation.

The input is an initial grid, if is rather unclear what is the definition of an initial grid? What is the suggested resolution in practice. How sensitive is the output why perturbing the initial grid? For example, when an user specifies the initial grid, which consists of many cells in my opinion, it is likely to be imprecise and inexact compare to the coarse grid of the underlying ground-truth. How robust is the approach when perturbing the input?


There are many inconsistencies between the main text and the supplemental material. For example, the main text states "which controls the generated surface via weighting the smoothness of neighbouring vertices". The supplemental material states that this is done using a graph neural network. It is unclear what is actually implemented? If the graph neural network is a major contribution, then the details should be in the main text. 

In summary, the paper takes the approach of first initializing a coarse grid using an implicit representation which is followed by learnable mesh subdivision. This approach is limited in the sense that a coarse implicit representation does not capture the right topology of the underlying model, which cannot be recovered using mesh subdivision. The paper does not compare against state-of-the-art implicit based representations (performed at high resolutions). It is unclear if similar results can not be achieved by these techniques. The paper is rushed, there are many inconsistencies between the main text and the supp. material. It is recommended to revise the paper and submit to future venues in vision and graphics.
  NA.

This paper proposes a new deep 3D shape representation that combines deformable tetrahedral grid, a learned SDF, differentiable mesh extraction, and learned subdivision. The model inputs a point cloud or coarse voxel grid and then interpolates SDF values learned on vertices of a tetrahedral grid (whose deformation is also learned), extracts mesh via deep marching tetrahedra, and applies learned a subdivision rule to obtain an output mesh. The network is trained by optimizing a loss evaluated on ground truth meshes as well as some regularization.

The authors provide comprehensive evaluation on the tasks of 3D reconstruction animals from coarse voxels as well as reconstruction from point clouds. They also include an ablation study.  This is a well-written paper that introduces a reasonable hybrid deep 3D shape representation. Representing the SDF on a deformable tet grid is a clever way to efficiently enable a coarse to fine representation, and ultimately extracting a mesh allows the model to be trained on mesh data with less artifacts from sampling or discretization. I also think that most of the experiments are quite comprehensive and show improvement over past work.

Although the proposed approach largely consists of combinations and perturbations of several existing methods, I do think that there is sufficient novelty. In particular, using the tet grid to represent SDFs and optimizing gradient flow based on relevant regions are reasonable choices that seem to help the overall method.

There are a few missing citations that should be discussed and do somewhat limit the proposed method's novelty. In particular, "Neural Subdivision" [Liu et al., SIGGRAPH 2020] is relevant in the context of the learned mesh subdivision step. "Neural Unsigned Distance Fields for Implicit Function Learning" [Chibane et al., NeurIPS 2020] and "SAL" [Atzmon et al., CVPR 2020] learn implicit fields using explicit geometry as input, somewhat undermining the claim that implicit methods require strict supervision in the form of occupancies or distances. Also, "Analytic Marching" [Lei et al., ICML 2020] is relevant in the context of differentiable mesh extraction.

I also have a question about Figure 6, which shows results on out-of-distribution voxels. The results are impressive, but only eight inputs are shown. How were these chosen? Are they representative, or do other specific inputs reveal failure cases?

Minor issues:

L44, 120: sign distance -> signed distance

L75,85,148,180: typology -> topology

L148: in practise -> in practice Sufficiently discussed in Sections 6 and in Supplementary Material.

DMTet aims to synthesize a plausible and detailed mesh from a coarse voxelized input (e.g. given by a user). It does so by regressing a low-resolution tetrahedral grid that is then refined and from which a mesh can be extracted via Marching Tetrahedras in a differentiable manner, which is the main contribution of this work. Using prior work, this mesh can be further subdivided in a differentiable manner before being used in the loss. The method is evaluated extensively on the TurboSquid dataset, including user studies. The experiments show significant improvements quantitatively and qualitatively over baselines.  The method is novel and the results are impressive. The main contribution, differentiable Marching Tetrahedras, as an alternative to Deep Marching Cubes, seems like it could be useful in other contexts as well. The evaluation is thorough and convincing. I believe it exceeds the threshold for acceptance in its current form already, although I have some suggestions for minor changes.

The exact contributinos are not clear to me. Which exact parts of Sec. 3.1 are novel? What are the changes made to [16]? Switching to SDF, adding volume subdivision (and making it differentiable), and using Marching Tetrahedra for surface extraction (and making it differentiable)? These should be clearly mentioned in the introduction.

The comment regarding prior work like Deep Marching Cubes in lines 147-151 is interesting. What is missing for full differentiability though are gradients into vertices (and associated values like position and features) outside the current tetrahedra, but I do not believe that this is a problematic limitation.

Current training time is a bit over half a week on 4 V100s. Is the implementation in any way optimized, e.g. are custom layers parallelized in CUDA? What speedups seem reasonably achievable with relatively little work? Which component is currently the major bottleneck?

The method is evaluated on only one dataset, which does contain fairly diverse animal shapes though.

The writing is fine but could use a final polishing touch.

I have several minor clarification questions that should be addressed in a revised version:

- What does the f(v_i) output of GCN in Eq. 2 mean? Does it overwrite the previous features f(v_i) from Eq. 1?
- How often is the Surface Refinement with Volume Subdivision (Eq.s 1 and 2) applied per forward pass? Only once or multiple times?
- Does the GCN for Learnable Surface Subdivision also take in the features f(v_i) from the previous step? If so, how are those features transferred from the tetrahedral structure to the extracted mesh? If not, why does Eq. 2 output f(v_i)?
- I assume the GCN uses Eq. 1-4 from the "GCN Model" paragraph in [10]? If so, it would be good for completeness if those equations are included in the supplement.
- According to lines 67-68 in the supplement, the subdivision step is applied "iteratively". Does that include the GCN that regresses v'_i and alpha_i, or only a repeated application of Figure B, where the Vs and alphas are regressed only once in the beginning and then the alphas are interpolated for all non-initial iterations? How many iterations are used?
- What effect does varying alpha have? Just very simply, e.g. higher alpha leads to smoother surfaces?
- What exactly is the small noise in line 74 in the supplement added to? The position of v? Before or after extracting the local SDF grid? And is the noisy position of v used when computing the SDF values (i.e. is the mesh actually slightly deformed) or does the noise only move the local SDF grid slightly around?
- How is the SDF function in Eq.s 4 and 5 in the supplement computed? Determine nearest face and take point-plane distance or point-triangle? Or something else?
- The sequence of surface refinement and subdivision steps that is mentioned in C.2 should be mentioned in the main paper already. 
- Line 116 and 118 have a typo "et.al." The limitations and future work are very well explained. I agree with line 377 and do not see potential negative societal impact.

Meta-review: This paper received mostly positive reviews, with the exception of reviewer f9sZ.  Taking into account feedback from the other reviewers as well as the message to the area chair, the AC tends to agree that f9sZ's concerns are not sufficient to hold back publishing this work.

The final revision of this paper should clarify issues of contribution/novelty brought up in several reviews and incorporate the additional experiments/fixes mentioned in the rebuttal posts.

Decision: accept