Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This submission proposes a differentiable importance sampling to estimate the rendering equation used in NeRF based methods. The method is derived using the re-parameterization method. The gradient descent is derived based on the re-parameterization method. Compared with the previous sampling method used in NeRF based methods, the proposed method leads to much lower sampling variance under the same computational cost. 

strengths_and_weaknesses: Strength

+The proposed importance sampling for Monte-Carlo (MC) estimate is technically sound by being based on the re-parameterization method. 

+The author also shows how to make the importance sampling along the ray differentiable by taking advantage of the re-parameterization. As a result, the new sampling method can be plugged into the neural rendering pipeline and make the implicit function trainable end-to-end.

+The proposed differentiable sampling method has much lower sampling variance as compared to the existing sampling method used in the NeRF-based methods.

+The density field is approximated by splines in closed form. This makes the integration along the ray for getting the light attenuation more efficient to compute, as compared to the previous methods where another integration is performed along the ray.

Weakness

-Although the proposed sampling method and its corresponding backpropagation process is sound and promising, the paper lacks the experiments to validate its effectiveness:

1.The method is only evaluated on a single scene (Lego)

2.The method is only compared to the base NeRF. There are works on speeding up NeRF without sacrificing performance during both training and rendering (eg. Plenoxel NeRF). How does this method compare those methods? And more importantly, can the method be plugged into those methods for further speed-up? Without the corresponding experiment, those questions can hardly be answered.

3.Key experiment missing for trade-off between speed and reconstruction quality: How is reconstruction quality degrades with the decrease # of samples (ie. speed-up) by using the proposed method?

4.Related to the missing speed-quality trade-off: From Tab.2, the estimation time per iteration  increases with the number of points in the splines and exceeds the baseline at around 16 points. So the question is how does the reconstruction quality compare with the baseline at around 16 sampling points. The paper does not include the quality comparison .


5.Comparison to other SOTA method missing: How is the proposed method compared to the SOTA NeRF methods in terms of reconstruction quality and computational and memory consumption 


-Paper needs more refinement: There are multiple question marks in the paper due to error in reference.


questions: For spline construction (L128-L130), what if the underlying density field is not piecewise linear, or have sharp spikes along a ray (eg. thin structure)? Will the spline approximation still be accurate enough to get opacity estimation?

L131-133: how to get the sampled grid t_0<...<t_m in the first place? Are they predefined? If so, how to guarantee that they don’t miss any non-empty space across different views and across different scenes? 
limitations: Experiment is performed on only one scene. So it’s unknown whether this method is finetuned for that scene or more general.

From L131-133, the sampling grid t_0<...<t_m cannot be guaranteed to cover the non-empty space.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper proposes a reparametrization of volume sampling for faster rendering of neural radiance fields. The key idea is that, when the density value (i.e., \sigma) is known at a few sparse points, volume rendering can be reparametrized for importance sampling, which reduces the variance and thereby accelerates the rendering of the neural radiance field. The paper provides the demonstration of lower variance (compared to uniform sampling), ablation study, and novel-view synthesis with a different number of samples both qualitatively and quantitatively.
strengths_and_weaknesses: While this paper tackles an interesting and important problem in neural rendering, which is an acceleration of NeRF rendering, it needs to be much improved in terms of presentation and evaluation.

- The proposed method is not evaluated properly. The key contribution of this paper is importance sampling that can accelerate NeRF rendering, but the provided evaluation does not demonstrate this contribution. Figure 2 shows the variance compared to uniform sampling, but the baseline should be the importance sampling used in the original NeRF. Any importance sampling should provide better results than uniform sampling, and this experiment cannot show any meaningful demonstration as the proposed method has an additional cost of density sampling in the previous step for the reparametrization. Also, in Section 5.2, the proposed method modified the original NeRF to reduce the computational cost of density sampling (by changing the density network from eight layers to three layers), which will reduce the quality of the density field and make the resultant comparison NOT apple-to-apple. Moreover, the evaluations are conducted only on a single scene (i.e., lego) and the improvement over the baseline is marginal even in the current evaluation.
- The presentation can be much improved. There are a number of typos and the notation is not consistent throughout the paper, which severely reduces the clarity of the paper considering the fact that the main contribution is in the reparametrization of equations. To list a few:
    - Line 47: R^3 → \mathbb{R}^3
    - Line 48: directionn → direction
    - Line 48: location x → \mathbf{x}, direction d → \mathbf{d}
    - Line 53: ray r → ray \mathbf{r}
    - Line 58: fauithfull → faithful
    - Line 75: c(o + Td, d) → c and d should be bold
    - Line 82: \sigma_r → both \sigma and r should be bold
    - Notation for Equations are not consistent; there are Eq n, Eq. n, n, Eq. eq. (n).
    - Lines 110, 123: Eqs. eqs
    - Line 130: r = o + td → r, o, d, should be bold
    - Line 208: Eq. ??
    - Line 224: Fig. ??
    - Line 225: ren dering → rendering
- The paper is based on Equation (2), which is the Riemann sum for the integral of volume rendering. However, it is not directly equivalent to the discrete volume rendering used in the original NeRF paper (Equation (3) in the original NeRF paper that is based on [Max 1995]) because of the small difference in the derivation (e.g., density is locally uniform between the samples). The difference in the equation should be clarified for a better comparison to the original NeRF.
questions: Why is the proposed method efficient compared to the hierarchical volume sampling of the original NeRF paper? There is no clear demonstration of this. Both require the coarse sampling stage, which is the same additional cost compared to the naive sampling strategy. If the proposed method modifies the network structure to reduce this additional computational burden, the original NeRF can also benefit from this modification. The main baseline should be the original NeRF paper, and the comparison should be stated much more clearly and fairly in the experiment.
limitations: The checklist says the limitations are described in Section 1, but Section 1 does not include any description of the limitation.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: A Monte Carlo sampling strategy is proposed for rendering NeRF-based density field representation. The method works by firstly densely querying the density field to compute an inverse CDF, which is later used to reduce the number of subsequent samples of colour network.

The method is compared to the NeRF baseline and achieved faster speeds (~20%) but at the cost of slightly worse image quality (~0.7dB reduction in PSNR).
strengths_and_weaknesses: The paper does address an important problem, that is to improve the efficiency of NeRF rendering. The proposed method is well motivated and indeed effective in that it does significantly reduce the sampling rate of colour network (8 to 32 samples per ray compared to 128 samples per ray in NeRF). However, on the other hand, it still requires a dense sampling of density network in the first stage, and as such the overall rendering efficiency is only slightly, if at all, improved without sacrificing performance. 

Many related papers are missing from related work section and should be cited and compared to, these include faster NeRFs:
- David B Lindell, Julien NP Martel, and Gordon Wetzstein. 2021. Autoint: Automatic integration for fast neural volume rendering
- Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. 2021. Derf: Decomposed radiance fields.
- Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. 2021. FastNeRF: High-Fidelity Neural Rendering at 200FPS
- Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. 2021. Plenoctrees for real-time rendering of neural radiance fields.
- Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. 2021. KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs. 

There are several papers on implicit surface volumetric sampling, that may be worth citing as well:
- Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction.
- Michael Oechsle, Songyou Peng, Andreas Geiger. 2021. UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction.

Experiments are too limited. There is only one case being the lego scene. Table 1 only includes two training sampling rates, what are the results for 1,2,8,16 samples per ray?

Some typos need to be corrected, e.g.:
line 71: propose a propose a
line 208, 224: broken ref

questions: I would suggest the authors to include missing references, and expand the experiment section.

To me the main drawback of proposed method is that dense sampling of density network still takes significant time. As such the speed gain is rather limited compared to vanilla NeRF. I wonder whether using a smaller density network can help in this regard.
limitations: I would suggest discussing the limitations in detail in section 6.
review_title: null

Meta-review: metareview: Reviewers noted that the paper contains some useful ideas for acceleration of neural rendering pipelines.  However, the paper as initially presented was rather preliminary with very limited evaluation.  The authors present considerably more material in the rebuttal, but as noted by the reviewers post-rebuttal, this extra material fails to demonstrate the computational advantage which is a primary contribution of this method.   It is hoped that the authors can continue this line of work, following the reviewers' suggestions, and demonstrate the method's value.   It may be that these ideas have other advantages, or that the advantage is ultimately less significant, in which case a more specific venue such as 3DV may be appropriate.

Although 2v6p did not provide a post-rebuttal comment, their initial review  was closely consistent with the others (albeit marginally more positive), so the post-rebuttal analysis of the reviewers and AC remains valid.

confidence: Certain
Decision: Reject