Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method for 3D scene reconstruction from a single image using implicit surface representations such as occupancy or SDF. The authors propose to incorporate loss functions on the spatial gradients to provide dense supervision in the 3D space in the case where 3D labels may be incomplete (e.g. open 3D meshes) or not well-defined everywhere. Experiments are performed on ShapeNet and ScanNet show that the proposed method can achieve competitive performance on single-image scene reconstruction tasks.
main_review: Strengths:
- I like the motivation of learning from 3D meshes that are not necessarily closed. This seems to allow 3D reconstruction neural networks to learn from a wider range of dataset resources, such as 3D scans or mesh reconstructions.
- The experimental settings are described in a very detailed manner with justifications. The results from the proposed method seem to show some improvements upon baseline methods, more notably in scene reconstructions.

Weaknesses:
- Although learning from either closed or open 3D meshes is an interesting motivation, this only allows one to learn SDF. One cannot define occupancy for open 3D meshes, and the example in Fig 3 is misleading. How would you define occupancy near open wholes? This part is unclear.
- The contribution is unclear and at most incremental. Two main components are described in the paper:
  1. Loss functions for learning occupancy or SDF (Sec 3.1). In my understanding, the only novel term is the spatial gradient penalty for occupancy. The rest of the two terms are standard loss functions for learning occupancy, so the additional first term is proposed. The "spatial gradient" term for SDF is precisely the eikonal regularization (Crandall & Lions, 1983), and the remaining terms are standard losses as well. (It is also unclear why the loss in the background paragraph is not incorporated).  
  The authors argue that conditioning the spatial gradients on pixels is novel. I think the authors should clearly state why conditioning on pixels is novel enough to be a standalone paper itself, as this to me is an overstatement and I don't see how simply conditioning for a different task is novel. Penalty on the spatial gradients have also been previously adopted for other single-image 3D SDF reconstruction tasks [A,B]. Also, the authors mentioned that "the spatial gradient $\nabla_{x,y,z}\hat{f}_\Theta(x,y,z)$ can be conveniently computed without the sampling procedure" -- how?
  2. The gradient expression of "spatial gradients" (Sec 3.2). My understanding is that this is basically treating the finite pixel differencing operation as a network op, which in nature is differentiable. I think this part is confusing in many aspects. First, I don't see why the authors emphasize that the formulation is closed-form, as the spatial gradient expression is clearly taken numerically and its derivative can be computed via automatic differentiation. If the authors meant to present the gradient of gradient expressions, please use second-order derivatives (e.g. $\frac{\partial^2 f(x)}{\partial x^2}$). It is also unclear how the 3D case (Eq 7) is derived; the authors merely presented an equation without elaborating its meaning. It is unclear what exactly $h$ and $w$ are; the authors referred to Fig 4(c) but there are no explanations in the captions either, which prevents a total understanding. Finally, a very important reference of Spatial Transformer Networks [C] is missing, as it was the first to advocate differentiable sampling.
- It is unclear what "engineering constraints" refer to (Fig 2 caption and before Eq 1).
- What is the relationship between $x,y,z$ and $i,j$? These notations are cross-referenced throughout the paper but their distinction was never explained. In Fig 4, are the feature maps sampled according to $x,y,z$ or $i,j$? What does $\nabla_{x,y,z}\phi_{i,j}$ in Fig 4(a) mean?
- Experiments:
  - The results from the proposed method has its own training recipe (architecture, optimization etc), and thus it is unclear where the better performance is coming from. (It could not be about the losses or spatial gradients at all, but a better design of architecture.) I think it is essential to see results where the baseline methods (e.g. OccNet, DISN) are retrained with the proposed losses incorporated. Would these methods yield a boost of performance?
  - Why are different baseline methods compared for different versions (low-res and high-res) of ShapeNet? I think it's sufficient to present just the high-res version, and have a more complete comparison with the baselines in the low-res table.
  - Fig 5: it is undefined what "positive/negative precision/recall" mean. How close is a surface/surfel prediction to the ground truth is considered a "positive precision"? The red/blue figures are meaningless without precise definitions.
  - Could the authors elaborate more what "amodal depth" means, why the surface are evaluated in terms of this new metric, and how they are visualized? Why is not the naive depth definition used to evaluate? In addition, if only depth were used to evaluate quantitatively, why would one care about reconstructing the entire scene, as one could alternatively go for a scene depth prediction task which yields better quality?
- There is no conclusion section. What have we learned about this paper?

Other minor problems:
- Please use only a set of notations for occupancy labels, not both {0,1} and {+,-}. Also, please do not mix the use of $\phi(I)$ and $\phi$, where the former should be a function and the latter a variable.
- Fig 6: why is the bed in AdelaiDepth only visible in view 0?

[A] Jiang et al. "SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization." CVPR 2020.  
[B] Lin et al. "SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images". NeurIPS 2020.  
[C] Jaderberg et al. "Spatial Transformer Networks." NeurIPS 2015.

summary_of_the_review: I think this paper has an interesting motivation of learning 3D reconstruction from incomplete raw 3D scans / open meshes, but there are major flaws in the method description and the experiments (detailed above). I also don't think there are either sufficient novelty or insights in the paper. I think the submission needs much major revisions with additional experiments to validate the effectiveness of the proposed DSG.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper describes novel loss functions for learning to predict an implicit 3D scene representation from a single image.  They argue that when working with real scan data of scenes (rather than single objects) it is difficult to generate accurate occupancy or signed distance function (SDF) ground truth as would be required for supervised learning.  Instead, they propose to only use occupancy or SDF supervision near the surfaces of objects; elsewhere, they rely on constraints on the gradient of the occupancy or SDF adapted from Gropp et al. 2020.

They perform a thorough evaluation on several benchmark datasets and compare against state-of-the-art competing methods.  They show that they outperform competing methods, even though in some cases their method has access to less supervisory data.    They also perform an ablation study to show the importance of various parts of the loss function.


main_review: Their proposed loss functions are novel as far as I know.  The idea makes sense, to only apply supervision near surface boundaries where the labels can be reliably produced.  They perform a thorough set of experiments including comparisons and an ablation study.  They also derive the closed-form gradients of their loss function and show the importance of using them over numerical derivatives.  Both the quantitative and qualitative results are convincing.

One question I had was why there were no coefficients to balance the strength of the regularization terms in equations (1) and (2).
summary_of_the_review: This paper has novel and interesting contributions to the field of single-image 3D reconstruction.  They provide convincing experiments to validate their contributions.  The paper is also well-written and nicely presented.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new method to learn implicit 3D scene reconstructions from single image input. The main improvement is a closed-form Differentiable Gradient Sampling.  By taking spatial gradient into  consideration, the proposed method can apply back-propagation of the  loss on spatial gradients to feature maps and allow the training for the case of without dense 3D supervision.   
main_review: pros: 
1. the formation of DGS is novel and interesting, with promising performance. 
2. detailed experiments on both shapenet and scan data
3. detailed ablation study 
4. better performance compared with pervious methods.

cons:
1. the overall learning loss is not novel (Eq 1 & 2), for the Eikonal regularization part.
2. there are some camera  parameters are involved in DGS, which is a bit hard to get in general.
3. it is not clear what is the percentage of the know voxel occupancy/SDF,   and how much would the rate effect the learning. It ill be better to have some discussion and  ablations. IMHO, generally if the value of voxels closed to the surface is known, it might be ok to learn with other regions are missing.
4. from the results (Fig9 &10), DGS  seems to be more likely to generate floating points, some discussion would be better to have here. 
summary_of_the_review: Overall, I think the proposed method is novel and with reasonable performance.  I am in favor for acceptance if the authors can provide some discussion about the cons listed above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a new method for single view 3D reconstruction. 
A conditional (image feature prior) implicit representation framework is proposed to reconstruct 3D scene from a single view.
In this paper, the authors propose that feature gradient is essential for watertight reconstruction and propose a differentiable gradient sampling method for the formulation.
Experiments have been performed on both synthetic and real datasets.
Superior results have been presented.
main_review: ## Strength
# Interesting and novel idea with the use of implicit representation
Implicit representation has been extensively explored in 3D object reconstruction and novel view synthesis recently. It is interesting to see how we can use implicit representation in various applications, e.g. single-view reconstruction.

# New sampling scheme for gradient
The authors have proposed a new sampling scheme for computing spatial gradient and thus a closed-form solution for loss propagation.
Although the approach is new, I still have some concerns listed below.

# Good result
The authors have shown good results in a variety of datasets, quantitatively and qualitatively.

# Adequate ablation study
An ablation study is also provided to support the effectiveness of the proposed method.


## Weakness
# Generalization evaluation
The authors provide a qualitative evaluation (single sample) on an unseen test image. It will be much better if more examples can be provided. Moreover, a quantitative evaluation will be more appreciated to show the generalization ability of the trained model.

# necessity of the gradient sampling scheme.
Though the proposed sampling scheme is new, I don't understand why it is necessary (i.e. eq 5). What will be the difference between this approach and a naive approach that compute a feature gradient map first, followed by simple differentiable sampling (eq.4).
Could authors shed some light on the difference? If the naive approach is a reasonable approach, why the proposed method is essential in this case?
summary_of_the_review: The paper is well written and presented overall.
The essential experiments are performed and the results are well presented.
However, I have a question regarding the major contribution (see weakness).
review_title: null

Meta-review: The paper presents a new way to train the prediction of implicit 3D scene representations from a single view. The main innovations are a novel numerically stable and memory efficient formulation of the derivatives of a loss function based on the spatial gradients of the implicit field, and focusing the training on regions near the surfaces of objects. The method leads to good performance, especially when training on imperfect ground truth scan data.

Concerns were raised about the novelty of the approach and its significance. These were adequately addressed in the author response and revisions. The experiments were found to be well described and executed, which increases the confidence in the approach and its potential impact. I recommend acceptance.
Decision: Accept (Poster)