Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a differentiable adaptation of the Poisson Surface Reconstruction algorithm for oriented point sets. It discretizes the Poisson equation over a 3D grid and use spectral methods to find the solution for indicator function values over this 3D grid. The final grid is fed to marching cubes algorithm to obtain the final mesh. The differentiability of the approach allows to use the proposed method in various optimization-based applications from which the authors consider optimization-based and learning-based 3D reconstruction tasks. Proposed models are evaluated on datasets with different level of details and noise present in the inputs and compared to both classical and more recent learning-based methods.
main_review: Pros:
* The paper is well-written.
* The proposed approach is novel and promising, both qualitative and quantitative results shown are strong.
* Evaluations are extensive and include both different datasets and different noise injection configurations. The authors additionally provide evaluations of the efficiency of the proposed method as well as some additional ablation studies.

Cons:
* The choice of some baselines for learning-based approaches is questionable. None of PSGN, 3D-R2N2 and AtlasNet models were designed for surface reconstruction from input unoriented points, so they were not optimized for this task. There is a least one very relevant work left unmentioned and uncompared to in this submission [1].
* It is clear that it is hard to provide a full mathematical explanation in the main text without using too much space but even in the supplementary materials some detail are missing. My concern is mainly equation 7 from the supplementary. Equations 5 and 6 partly justify it but do not account for the appearance of $\tilde{g}$. It is said that it is introduced to mitigate Gibbs phenomenon, but it is not obvious why the addition of $\tilde{g}$ to the solution of Poisson equation in frequency domain do not corrupt this solution (there will not be an identity if one put $\tilde{\chi}$ in FFT of Poisson equation).

Overall, I believe this is a strong submission. The cons should be addressed but, in fact, are minor in my opinion and should not justify rejection.

Minor comments:
1) L146-148 "$\tilde{g}_{\sigma, r}(u)$..." - rephrase the sentence, something is missing.
2) L183 "a a"

[1] Erler, P., Guerrero, P., Ohrhallinger, S., Mitra, N. J., Wimmer, M. Points2surf: learning implicit surfaces from point clouds. In ECCV’20.

***
After reading the rebuttal and other reviews I decided to improve my rating since my concerns were answered. This is an overall strong submission with great novelty and promising experimental results despite the limitations of cubic complexity for indicator grids.
limitations_and_societal_impact: The authors do not provide any discussions about these impacts besides stating that their approach shares the risks with all the other learning-based 3D reconstruction approaches.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This work proposes a novel shape representation using point clouds. This is enabled by unique differentiable poisson solver layer that enables to convert the oriented point clouds to full volume representing the shape. I think the main contribution of this work is the spectral technique that solves poisson surface reconstruction problem, which can be efficiently integrated into neural workflows. While this work mostly focuses on surface reconstruction, I believe this representation can become broadly applicable to various problem domains.
main_review: This paper has both high-level novelty ("shape-as-points" representation) as well as low-level technical contribution (spectral differentiable poisson solver). I think the proposed representation is valuable and will inspire future work. The evaluation is thorough and the method offers several quantitative advantages over state of the art methods (in terms of runtime and quality). I would be excited to see this work at NeurIPS. 
limitations_and_societal_impact: Limitations are adequately addressed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This work introduces a differentiable points (+normals) to mesh layer based on a differentiable formulation of Poisson surface reconstruction.
First, the proposed method is validated in point cloud optimisation tasks where the objective function is expressed w.r.t surface meshes.
Then, the method is proposed as a way to represent 3D shapes with oriented point cloud in a learning setting.
Crucially, this results in a shape parameterization, dubbed Shape As Points (SAP), that is more lightweight and faster than neural implicit fields, while still yielding watertight shapes and being able to handle arbitrary topology. 

main_review: Strengths:

- GPU-accelerated differentiable pipeline to solve Poisson equation could be useful to research community at large 

- New shape parameterisation has similar properties to neural implicit fields (can handle arbitrary topology, produces watertight surfaces) while being more lightweight and efficient

- Extensive experimental evaluation demonstrate the advantages of the proposed representation when 3D supervision is available 

Weaknesses:

- Limited resolution: the proposed pipeline is, by design, limited by the cubic memory requirement at training time. This is not the case for neural implicit fields, and in some sense the proposed representation is a step-back from them, which allowed to represent arbitrary topology while breaking free from memory intensive 3D grids.

- 2D supervision: the proposed approach is not evaluated in a context in which 3D supervision is not available, although, in theory, any off the shelf differentiable renderer would be compatible with the pipeline. Without this evaluation, it is hard to say if the proposed approach has any fundamental limitation in a differentiable rendering context (e.g. due to its underlying sparsity?).

Question for the authors:

along lines 45~48 authors claim: "It is easy to initialize SAP with a given geometry such as template shapes or noisy observations. In contrast, neural implicit representations are harder to initialize, except for few simple primitives like spheres". The claim is re-stated in table 1. I don't understand what do authors exactly mean with this: 
In the learning setting, where oriented clouds of points are used as underlying 3D shape representation, we do not really care about initialisation. The same though holds for neural implicit fields, as the weights of the underlying MLP are similarly initialised using standard techniques. 
In the optimization setting, although I understand that the SAP framework would allow for initialising the shape using noisy points and normals, the reported experiments still decide to start from a sphere (see supplemental video), probably because this results in a better local minima attained by the optimisation process. Note that a sphere is indeed a trivial initialisation for  an implicit field, as correctly stated by the authors along lines 45~48. 
In light of all this, I think this claim should be either better argumented/explained or toned down.
limitations_and_societal_impact: As stated above, I think the cubic memory requirement at training time should have been better pointed out in the paper.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper formulates a differentiable Poisson Surface Reconstruction layer, which maps an oriented point cloud to an indicator function over the interior of the object. The Poisson solver is done in the spectral domain on the GPU for computational efficiency. It shares many of its advantages with other neural implicit representations, but because it produces an indicator function over the whole space, there is no need to query a network over an entire voxel grid. 

The layer is incorporated into two tasks: an optimization-based 3D shape reconstruction that fits a spherical point cloud initialization to a target point cloud, and a learning-based 3D reconstruction task that matches noisy point cloud input to a ground truth mesh. The non-differentiable step of Marching Cubes in these pipelines has its gradient approximated with the inverse surface normal, which is quite natural. The experimental results seem good, and beat out the existing state-of-the-art, and an ablation study is performed as well on the learning-based task to investigate the form of the feature extraction network there.
main_review: I think the method is an interesting and novel implementation of the PSR algorithm, used to produce a better implicit representation than the existing state-of-the-art methods. I'm generally convinced by the experimental results, though I would appreciate some more information on the structure of the offset and normal prediction network. It was not quite clear to me why the particular architecture choices were made and whether other options were considered. The ablation study sticks to the convolutional point cloud encoder that they noted earlier, and considers two different versions, but does not consider other network architectures.

With respect to the layer, I was also curious to know about how much spectral resolution was used in the Poisson solver, and how the sigma was chosen for the Gaussian smoothing kernel.
limitations_and_societal_impact: I think the authors did a fair job of addressing these points.
review_title: null

Meta-review: Congratulations, the paper is accepted to NeurIPS 2021!
Please incorporate additional experiments, edits and corrections as discussed in rebuttal. 
Decision: Accept (Oral)