Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose an approach (DMTet) to obtain detailed 3D shape from coarse voxels/pointclouds.
Given an input DMTet predicts SDF values at discrete tetrahedral grid. Authors use this SDF to obtain a coarse surface which is further used by GCN to refine the tetrahedral grid. Surface is then recomputed. Authors use a second GCN to predict parameters for loop subdivision to further increase the mesh resolution. 

main_review: + The demonstrated results look good.
+ Authors include detailed evaluations in supplementary which are appreciated.
+ Paper is well written and easy to follow.

A few comments:
1. Supplementary, Sec E.2 Table B. Why do the numbers for point cloud reconstruction on ShapeNet for baselines look different than Table 1 in [16]? Was the experimental setup somehow different?
2. Eq. 1,2: f(v_i) is input to the GCN, what is the advantage of reconstructing it back?
3. L286-296: Numbers could be put in a table for easier parsing.
4. Fig 7: I find it funny that the lamp shown in the second row to showcase superior performance of the method is also used in supplementary fig. J to show the limitations of the method.

Animals and shapenet objects are relatively smooth shapes. Can the proposed method be used to generate detailed shapes such as clothed 3D humans? This would be a challenging testbed to assess the gains from the proposed surface refinements (adversarial, volumetric and surface).
limitations_and_societal_impact: Authors discuss limitations of their method and broader impact in the paper.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a method for synthesizing 3D shapes from coarse voxels using a 3D conditional generative model. 
The main *claimed* contribution is a differentiable marching tetrahedra layer that can convert the implicit signed distance field to the explicit surface representation. The paper also claims that with such a differentiable marching tetrahedra (MT) layer, one can jointly optimize the surface geometry and topology during training. 
Despite the fact that the presented results do look promising compared to previous works, I doubt that the improvement comes from the proposed differentiable MT layer, due to the fundamental technical flaws which I will elaborate on later.
Instead, I believe a great part of the performance comes from the direct and full SDF supervision provided in the seemingly minor *regularization* term and the deformable tetrahedral grid, which the authors fail to provide ablation study on these two algorithmic components.  

main_review: - The technical contribution of the proposed work is very limited. The paper claims that the main contribution comes from the combination of the deformable tetrahedral grid and the differentiable marching tetrahedra (MT) layer. Since the deformable tetrahedral grid is an existing technique, the only new algorithmic component is the differentiable MT layer. However, in this work, they actually directly apply the traditional MT algorithm for converting the signed distance field to mesh representation, with *NO* modification. Though the work also incorporates volume and surface subdivision techniques, these are very common approaches for coarse-to-fine reconstruction. 

- There is a fundamental flaw in claiming the original marching tetrahedra algorithm is differentiable. As agreed by the authors, there is a singularity in the original MT algorithm -- when the signs of the edge endpoints are the same, there are no gradients flowing into this edge. In addition, in line 218 to 220, the paper also admits that only the vertices that are close to the iso-surface in the tetrahedral grid receive gradients, while the other vertices do not. Hence, I do not think it is fair to call the proposed MT layer (actually the original MT algorithm) differentiable. 

- The writing is very misleading, as it tries to convince the readers that the main improvement comes from the differentiable MT layers (the main contribution of the paper). However, as analyzed in the above point, due to the fundamental flaw in the differentiability of the MT layer, it is very unlikely (if not impossible) that solely using the proposed MT layer can achieve a large topology change to generate high-fidelity results. Hence, I do not believe the proposed MT layer contributes a great part in achieving good performance in the results. Instead, the performance boost should probably be credited to the deformable tetrahedral gird (an existing technique) and the full SDF supervision in the so-called *regularization* term. However, the paper fails to provide any ablation study regarding the effect of these two components. 
I would like to see the performance without the SDF regularization term, the deformable tetrahedral grid, and both. Otherwise, I do not think the evaluation is comprehensive.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method to generate fine-grained shape from coarse volume or sparse point clouds. The pipeline starts from a feature extractor that produces a feature volume from the input, which is then used to predict and refine sdf values on a pre-defined  tetrahedral grid. Both the vertex location and sdf are then updated, with surface subdivision to capture more local features. At last, a differentiable marching tetrahedra is run to get the mesh. The model is supervised by both Chamfer distance on points sampled from the surface, as well as the SDF of tetrahedra vertices. The overall performance on shape synthesis with coarse volume or point cloud achieves the best quantitative performance compared to the baselines, and the advantages of MT over marching cube are analyzed.
main_review: Strength:
- The use of deformable tetrahedra seems to be an elegant, i.e. accurate and efficient, representation for 3D shapes.
- The performance on both tasks clearly outperform baseline methods, and the qualitative results looks promising.

Weakness:
- The task achieved by this paper is 3D super-resolution rather than shape synthesis. The pipeline creates a one-to-one mapping between the input and output, and therefore I don't see how novel shape could be synthesized.
- The deformable tetrahedra is a generic representation not only just valid for the proposed tasks. To fully demonstrate the effectiveness, other common tasks, e.g. shape from image, reconstruction, could be investigated.
- It might be good to have some tight comparison to the marching cube based pipeline under two tasks. I see both of the tasks could be similarly achieved with regular cubical grid plus deep marching cube [26]. It is good that DMC [26] is compared in table 3. However, one might be interested to see how the pipeline work with exactly the same pipeline using cube instead of tetrahedra. In this way, we would be able to clearly see the advantage of using tetrahedra.
- When compare MT and MC, run-time efficiency should be provided.

Overall, this work provides an extension of the marching cube based approach with tetrahedra. The major novelty seems to be more engineering oriented (if I didn't miss anything important), and adding more experiments might be helpful to highlight the importance of the contribution.
limitations_and_societal_impact: No comment.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes deformable tetrahedral grid for 3D reconstruction. It aims to combine the advantage of both implicit surface and explicit surface by extracting the mesh using a differentiable marching tetrahedra layer and refine it using a grach convolutional neural network. The proposed method is evaluated an animal dataset collrected by the authors and ShapeNet objects, taking either a low-res voxel grid or a point cloud as input.
main_review: ### Strengths
1. The paper is well-written and easy to follow.
1. It is interesting to define an adversarial loss on local regions for refining geometry.
1. The authors motivate the choice of MT instead of MC using a toy example, which I found interesting and convincing.
1. The proposed method demonstrates superior performance on 3D reconstruction of single objects, in particular for thin objects.
1. User studies are conducted to evaluate the quality of the 3D reconstruction of the animal dataset.

### Weaknesses

1. The proposed method is fairly similar to DefTet. As I understood, the major differences compared to DefTet are: a) using SDF instead of occupancy, b) hierarchical sampling, c) using GCN to predict vertice offsets and d) adversarial loss. Despite achieving better performance, these improvements appear to be incremental conceptually. Moreover, DefTet conducts experiments on Novel View Synthesis and Multi-view 3D Reconstruction, while the proposed method is only evaluated on 3D single object reconstruction from 3D input. What is the reason for this? It would be great if the author can elaborate more about the key differences compared to DefTet in the paper.

1. Considering the differentiability of MT/MC, I believe that it is also related to the loss. If there is a direct loss defined on the SDF field ($L_{sdf}$ as in this paper), I agree with the authors that the original MT/MC can be applied directly. However, it is not straightforward for me how the topology can be changed freely when only chamfer distance loss ($L_{cd}$) is considered. For example, it might be that $L_{cd}$ is not monotonically decreasing when transitioning from the initial topology to the target topology. Note that this is not a problem when using $L_{sdf}.Some more specific suggestions/questions regarding these concerns:

    a. In the main experiments $L_{sdf}$ is always applied. But the experiment of Fig.A in suppmat uses $L_{cd}$ only. While it is interesting, it is not straightforward to me how the original MT changes topology. It would be great if the authors can demonstrate the topology change of one single tetrahedron when using only $L_{cd}$. For example,  one can show how the topology changes from the left topology to the middle one, and from the middle to the right in Fig.3.

    b. The authors also argue that the refinement step using the GCN module is a reason that allows for topology change (L180). As I understood the topology change depends on the supervision signal (and hence the gradient) instead of the network architecture. It would be great if the authors can elaborate a bit more about this.

1. The idea of using an MLP to predict the implicit surface and extracting the explicit surface such that the loss can be directly applied on the surface is conceptually similar to MeshSDF [38]. It would be good if the author can explain why a comparison to MeshSDF is not considered. 

1. While it is great that the authors provide ablation studies to verify some of the design choices, I found the ablation of some losses is missing (e.g. $L_{sdf}$ and $L_{def}).

1. The authors only consider experiments on single objects. It is unclear whether the method can be applied to room level (like in ConvONet).

1. This is not a major concern but the training scheme appears to be quite complicated with multiple steps and takes a very long time (6 days on 4 NVIDIA V100 GPUs).

In summary, I found this paper well-written and interesting but I have some major concerns regarding comparisons to existing works and the differentiability of MT/MC. I would like to increase my rating if the authors can address my concerns.

### Update after rebuttal

The authors have addressed my concerns regarding the differentiability of MT/MC. Therefore I vote for accepting this paper.

### Detailed comments

1. It would be good if the authors can provide the motivation for choosing the LSGAN objective instead of other GAN objectives.

1. Some of the citations are not up-to-date. For example, [6] and [37] are both published at conferences while they are still referred to Arxiv.

limitations_and_societal_impact: The authors provide failure cases in the suppmat and discuss the broader impact of their work.
review_title: null

Meta-review: The paper initially received mixed reviews. After the rebuttal, three reviewers were positive while one remained negative. The shared concerns include the limited technical contribution, the missing ablation studies on $L_{sdf}$, and the lack of a comprehensive evaluation between MT and MC. The AC shares these concerns.

But the AC also agrees with the reviewers that all engineering efforts in making the pipeline working and achieving good results, as well as the potential impact to the community on providing an easy solution to directly adopting MT/MC in a neural network, are useful contributions. The AC, therefore, recommends acceptance of this borderline case. The authors are strongly encouraged to include the missing ablation studies and comparisons with MC in the camera ready.
Decision: Accept (Poster)