Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper is based on the "sign agnostic learning" (SAL) method for capturing signed distance functions with neural networks. It extends this method by incorporating derivative information, which interestingly can likewise be handled in a sign agnostic manner. (Maybe I missed this somewhere, but if the derivatives are sign agnostic, couldn't it happen that the inside is positive? Did the authors encounter that in some cases?)

The paper presents and motivates this extension together with an additional theoretical insight about the minimal surface property of SAL and SALD. In line with SAL, the paper presents a nice variety of results for shapes from different shape databases. The quantitative results are also convincing. It's interesting to see the substantial difference between the VAE and AD architectures. For the comparison with SAL it's good to see the direct improvements from the derivative loss with a VAE.

The paper leans heavily on SAL, and the change in terms of the overall method seems to be fairly small. Nonetheless, I think it's an interesting insight that the sign agnostic derivatives can be included in this way, and I found it interesting to see how much they improve the results.

Given that learning signed distance functions is a very active topic, and a very useful building block for a variety of adjacent works that use learned SDFs, the proposed SALD approach seems like a very nice advancement of the state of the art.

So, overall, I really liked the paper. Figure 2 alone is impressive, and makes a good case for the method. Together with the nice presentation and set of results I think this paper makes for a very good addition to ICLR.

review_title: SALD review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 
This paper presents SALD, a new type of implicit shape representation that, in addition to predicting the signed distance function, aligns the gradients of the distance function with that of the neural distance field. The resulting algorithm, for example, has improved approximation power and better preserves the sharp features than the ancestor SAL (sign agnostic learning). The formulation is such that the architecture can consume raw point clouds. 

STRENGTHS

This paper certainly speaks to me. First of all, learning implicit representations directly from raw point clouds can allow for interesting applications such as better generative models or efficient 3D reconstruction networks. The approach is very sensible. In fact, aligning gradients of the implicit surface with the ones of the data is not a new idea and has been done for instance in quadric fitting:
* Birdal, T., Busam, B., Navab, N., Ilic, S., & Sturm, P. (2019). Generic primitive detection in point clouds using novel minimal quadric fits. IEEE transactions on pattern analysis and machine intelligence, 42(6), 1333-1347.
* Tasdizen, T., Tarel, J. P., & Cooper, D. B. (1999, June). Algebraic curves that work better. In Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149) (Vol. 2, pp. 35-41). IEEE.

[the paper might benefit from including those especially because it has related work sections called 'primitives' and 'implicit representations'.]. 

This is not a drawback but just the opposite: there is a strong prior evidence that such approaches are useful. I also like that the authors spend a reasonable amount of effort for theoretical analysis. Though, I believe that this can be extended to more realistic scenarios (as the authors aptly explained in the limitations).  


WEAKNESSES / ISSUES 

- In addition to aligning the gradients, many works benefit from constraining the gradient norm of the implicit function be |\nabla| = 1. See for instance:
* Slavcheva, Miroslava, et al. "Killingfusion: Non-rigid 3d reconstruction without correspondences." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.

Can we think of a similar approach here? Could the paper show some ablations with regularizers concerning the gradient norm?

- Nowadays, the use of implicit 3D representations is omnipresent. In the evaluations, would it be possible to compare against the variants of DeepSDF (e.g. Curriculum DeepSDF or MetaSDF etc.)? With that, it might also be nice to include some more qualitative results in the supplementary. 

- Would it be possible to include additional real objects that are non-humans? This might involve for instance cars in an autonomous driving scenario.

- Some discussions on the following aspects could be valuable for the reader: (i) What would be a good suggestion to handle thin-structures? It seems to be a common issue among many SDF-like methods. (ii) The use of raw point sets is good, but such data usually come partially observed. Could this method support partial observations? If not, could there be workaround?

- The Chamfer distance and the variations thereon are obviously not well suited to assess the accuracy of the deep implicit representations. This creates an urge for better quantitative metrics, maybe the data driven ones. For the future, I would strongly suggest thinking about those to have more meaningful evaluation data.

- Some minor remarks:
* Can we already compare D and D' and give an intuition about what they might refer to at the place they are first defined?
* "they strives to" -> they strive to
* "tested SALD ability" -> tested SALD's ability
* "the surfaces produces" -> "the surfaces produced"
review_title: A good paper addressing an important problem.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies how to generate meshes from raw point clouds. In particular, this paper proposes a framework which is built on top of recent "sign agnostic learning (SAL)" work. Compared to SAL, this work adds a gradient penalty term, which encourages the derivative consistency. The problem studied in this paper is important, however, the proposed method is very incremental and has several motivation issues. I summarize the pros and con as follows.

Pros:
1. The idea of using gradient penalty to learn "sharp" signed distance function seems convincing. In Figure 4, the proposed method preserves sharp features compared to its counterpart SAL.
2. This paper presents a theoretic intuition why SALD works -- under uniform distribution assumption, SALD finds the global minimum. 

Cons:
1. My biggest concern is the motivation to learn sign distance function from its unsigned observations. For data (ShapeNet and FAUST) used in this paper, signed distances are immediately available -- one can easily convert a mesh to its implicit representation. To me, learning signed distance function (as DeepSDF does) is more convincing since the direct supervision is available. So why does this method bother to learn the proxy objective (unsigned distance function)? 
2. Following 1, the most obvious application of this paper would be learning signed distance function when the distances are not available -- the input is either LiDAR scan or depth image. In that case, if the paper can reconstruct realistic 3D models, it will be much stronger. 
3. To some extent, this paper uses neural networks to learn sign priors from data. There are multiple existing works on this direction which this paper doesn't mention (or briefly mentions but doesn't compare to). E.g, "Deep geometric prior for surface reconstruction" and "Point2Mesh: A Self-Prior for Deformable Meshes". The paper should at least explain the differences of the tasks if it doesn't compare to them. 
4. In the implementation detail, the paper says it uses a similar architecture to DeepSDF in the auto-decoding case. However, the method shows improvements over DeepSDF. This seems impossible given that DeepSDF learns from direct signed distance supervision. So I am wondering if this is due to model size difference. I'd like to see more comparisons to DeepSDF under exactly the same model capacity. 
review_title: Good problem, weak motivation, and issues in experiment design. 

final_score: rating: 7: Good paper, accept
review_content: ## Summary of paper and contributions
SALD extends prior work on Sign Agnostic neural implicit shape representations to include a loss term on the derivative of the implicit function. The authors justify the benefits of derivatives in 2 ways: (a) By citing prior work [1] which shows empirically that derivatives decrease sample complexity of deep ReLU networks, and (b) By showing qualitative improvements over SAL without derivatives.

The authors show qualitative evidence that global minimizers of sign agnostic losses (with and without derivatives) satisfy the *minimal surface property*, a desirable property of solutions in commonly discussed the surface reconstruction literature. They demonstrate this property via 2D experiments and via a motivating theoretical example. 
Finally, the authors show their loss function can be integrated into existing generative shape modelling pipelines, comparing results on ShapeNet and D-FAUST against DeepSDF which requires pre-computed SDF data, and SAL which can operate on raw inputs.

## On the benefit of using derivatives
The authors cite [1] to motivate the benefit of including derivative terms in the loss. In the case of deep ReLU networks such as the one used by the authors, this prior work shows an empirical reduction in sample complexity when regressing low dimensional functions (Section 4.1) motivated by a theoretical intuition (Section 3).  While the neural implicit functions learned by SALD are indeed low dimensional, the shape-space learning problem is not: It learns a map from a point set (consisting of many points) or a high dimensional (256 in the SALD case) latent code to an implicit function. Given this, I don't believe the authors can simply claim a reduction in sample complexity by citing [1] without demonstrating further experimental evidence, especially given the fact that the experiments in the paper do not show SALD drastically improving over SAL.

In particular I would be more convinced by an experiment showing the degradation of SAL vs SALD as the number of available samples for a shape is decreased when (a) regressing a single shape directly from data (such as in IGR [2] Section 6), and (b) regressing a shape using an auto-decoder. 

## Minimal surface property
Showing that global minima to SAL may satisfy the minimal surface property is indeed quite interesting. I do feel however that the claim in the paper regarding this is a bit oversold. In particular "We prove that SAL enjoys a minimal length property in 2D" (Abstract) and "Identifying and providing a theoretical justification for the minimal surface property of [sal]." (end of Section 1). The minimal surface property is well known in the surface reconstruction literature (e.g. [3] cited by the authors in Section 3) and the theorem shown by the authors appears to be for a specific case in 2D unless I am missing something. While these results are not trivial, I feel the contribution should be rephrased to something along the lines of
"We give empirical evidence and theoretical motivation that minimizers of SAL-type losses produce solutions satisfying the minimal surface property [citation]"

## Experimental Evidence
I feel like the choices of datasets and baselines are sufficient to show the effectiveness of SALD. There are two experiments however which I feel are missing from the paper:
 1. The sample complexity experiment described above.
 2. Some kind of performance evaluation. I imagine that computing losses on gradients of networks is quite expensive. How much is the increase in runtime compared to the gains in accuracy?

## Summary of review
Generalizing SAL to include derivative quantities is a natural next step for this line of work. The authors show that SALD improves performance over the state of the art on Shapenet and performs comparably on D-FAUST. While these results are great, I feel the paper is missing a few key experiments described above, and that the claims around the minimal surface property are a bit overblown. I am rating this paper as marginally below the acceptance threshold but am more than willing to increase my score if the authors make the requested revisions or give a strong justification as to why they are unnecessary in their rebuttal. 

## References
[1] Czarnecki et. al. - Sobolev Training for Neural Networks
[2] Gropp e.t. al. - Implicit Geometric Regularization for Learning Shapes
[3] Zhao et. al. - Fast surface reconstruction using the level set method
review_title: Good paper but needs some revisions for acceptance

Meta-review: Congratulations!  The reviewers unanimously viewed this work positively and were in favor of acceptance to ICLR.

While the current revision already addresses many reviewer concerns, it may be worth adding some of the datasets pointed out by R3 or comparing to some of the papers suggested by R1.
Decision: Accept (Poster)