Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a novel neural point-based rendering framework that can achieve view synthesis quality comparable to volume rendering based methods while being 100x faster. The method is based on existing approach but with non-trivial novel techniques that significantly improve the rendering quality, including SH features, point dropout layer, 2-D rendering without batch norm, point pruning and growing. Experiments on various 
strength_and_weaknesses: Strengths of the paper:
1. The paper is well written and easy to follow. 

2. The contribution of the paper is clear and well-presented by authors. While this paper is built on existing works, author did a good job to combine them together and also proposed non-trivial implementation improvements that lead to higher quality results, which may be very useful for future researchers. 

3. Experiments are convincing and comprehensive. Authors test their method on many widely used datasets. 

Weaknesses of the paper:
1. The technical novelty may be somehow limited. But I believe this is not a very big issue. 

2. Spherical Harmonic Point Feature: I have two questions here. First, if we only have degree 2, how could we handle very specular materials in DTU dataset? Second, why s_i's dimension is K/9? In which step do we turn feature vector to actual SH coefficient and compute the radiance? 

3. Point Dropout Layer: Will random subsets cause flickering in different views? From the video, this does not happen. Do authors have any comments on why it won't happen? 

4. Point sculpting: Can authors comment on how the proposed point pruning and adding method are different from those in Point-NeRF? This may help readers better understand the relationships between the 2 methods. 
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clearly written and easy to follow. 

Quality: The proposed method achieves view synthesis quality comparable to state-of-the-art while being much faster, which I suggest is good enough to be accepted. 

Novelty: While the proposed method is largely built upon existing works, as pointed out by authors themselves, many new implementation improvements may be very useful for future researcher who hopes to use point-based representation. 

Reproducibility: The code is attached to the submission. 
summary_of_the_review: I believe this is a solid paper with clear contribution and improvements. While the technical novelties may be limited, a significant faster point-based neural rendering framework with high-quality can be empirically very useful. Authors also did comprehensive and convincing experiments to demonstrate the improvements of the proposed method. Therefore, I lean towards accepting this paper. I will really appreciate if authors can address my question in the weakness section. I will also learn from other reviewers's comments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a point-based rendering framework for novel view synthesis. The method starts with an initial point cloud estimation from MVS systems. With the proposed key technique named point sculpting, the framework is less prone to initial point cloud error and could add and remove points within the optimization process. The method uses point rasterization instead of volume rendering when synthesizing novel views. The paper includes extensive experiments showing good results.
strength_and_weaknesses: Strength:

- The proposed point sculpting technique is novel. The method achieves good performance on all benchmarked datasets and has faster training and inference speed when compared to NeRF or other point-based baselines.

- Using spherical harmonics in the feature space and eliminating the need for a dir encoding MLP seems to be a novel but useful technique.

Weaknesses:

- According to Appendix B.2, the authors applied different MVS networks (at least different weights) when evaluating on different datasets. Especially, when evaluating on LLFF data, they finetune the MVS model scene-by-scene. This may cause concern/confusion because 1) the time for COLMAP and scene-by-scene fine-tuning should be taken into account when comparing, rendering the method less efficient for these scenes; 2) it is unclear why not using COLMAP point clouds directly. The authors should clarify these and ideally use a same generalized MVS model for a fair comparison.

- Pulsar is a very related direct baseline of this method. However, a comparison with Pulsar seems to be missing in this paper.

- The intuition behind employing a U-Net for rendering is unclear. Is it possible to use per-pixel MLP to render the features (MLP: feat -> RGB)? Is the U-Net pre-trained and weight-shared across datasets?
clarity,_quality,_novelty_and_reproducibility: This paper is good and clear in writing. The ideas proposed are novel. The authors have provided abundant information to reproduce the work.
summary_of_the_review: Overall the method proposed is novel and effective. I am leaning toward acceptance due to their technique contributions and good performance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper addresses the problem of novel view synthesis from a set of input images. The authors presented a point-based method which is build on top of exsiting works on differentiable point-based rendering. They introduced point sculpting for pruning and adding points to improve the photo-consistency and completness of the reconstruction. They also exploited sphereical harmnonics in high-dimensionalpoint feature space to capture non-Lambertain visual effects. They proposed a point dropout layer that can significantly improves the generalization to novel views. Experimental results show that their method can achieve better visual quality than the popular NeRF approach and at the same time being 100 times faster in rendering speed.
strength_and_weaknesses: Strength
+ The proposed point sculpting can effectively improve the completness of the reconstruction.
+ The proposed point dropout layer can alleviate the over-fitting problem by giving all points a chane to get optimized even if they are covered by others in the input views.
+ The proposed high-dimensional spherical harmonics point features can effectively capture non-Lamberian visual effects.
+ The proposed method is 100 times faster than the popular NeRF approach in rendering.
+ The proposed method can produce better visual quality than NeRF and other point-based methods.
+ The proposed method achieves SOTA in terms of SSIM and LPIPS metrics.
+ The proposed method allows easy fine-grained scene editing due to its point-based representation.

Weaknesses
- Examples in the Appendix show that the proposed method is good at capturing complex textures and strong non-Lambertian effects, but tends to produce over-smoothed results for tiny structures. This may suggest that the proposed spherical harmonics point features are effective in modeling the view-dependent appearance of the scene but the point-based representation may not be good (dense) enough to  capture fine scene geometry. The over-smoothed results may also be related to the use of U-Net for final rendering. Finally, the averaging of 2 results generated from 2 random point subsets may also contribute to the problem. More analysis should be carried out regarding this issue.
- In 3.2, it was mentioned that U-Net was employed to convert the 2D feature map F of a target veiw into RGB image. However, in 4.2 (point adding step), it was mentioned that the point features were treated directly as RGB values during rasterization without using U-Net in optimizing the existing point.  Please clarify how can the features be treated directly as RGB values in this step.
clarity,_quality,_novelty_and_reproducibility: Overall this paper is clearly written and easy to follow. The proposed point sculpting, though sounds a bit engineering, is effective in improving the completness of the reconstruction. The proposed point dropout layer is also sound and novel, and is effective in improving the generalization to novel views. SOTA results have been reported. Sufficient details have been provided for reproduction of the results.
summary_of_the_review: The improvements introduced to the point-based approach for novel view synthesis are novel and effective. Comprehensive evaluations are presented, showing SOTA results over existing NeRF and point-based methods. I would support accepting this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper receives 2x accept, good paper and 1x marginally above the acceptance threshold. The reviewers agree that the proposed method leads to higher quality results, the paper is well written and the experiments are convincing and comprehensive. The weaknesses are mostly clarifications that are answered in the authors' responses. 

justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: This paper receives 2x accept, good paper and 1x marginally above the acceptance threshold. All reviewers think that the proposed method is effective and the results are good.
Decision: Accept: notable-top-5%