Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper (S-NeRF) aims to train a NeRF using images from driving datasets (e.g., nuScenes and Waymo) with a limited number of views (2-6 images). The improvement over Urban-NeRF is that it densifies sparse LiDAR depth for depth supervision, through a reprojection-based confidence map approach. The main claimed advantage is robustness to noise in the depth point cloud. The experiments show sharper, more recognizable novel view synthesis output, over Urban-NeRF and Mip-NeRF.
strength_and_weaknesses: Strengths

- Achieving robustness to noisy depth in this setting is a practical goal.
- Using depth completion priors and multi-view reprojection is a reasonable approach.

Weaknesses

- I suspect the improvement is due to the separate evaluation of static background and moving foreground objects (how was this done for the baselines?), on top of the heavy usage of the pre-trained NLSPN depth completion network.
clarity,_quality,_novelty_and_reproducibility: (minor concerns) My impression is that the paper is hard to understand and not polished enough.

- Writing quality seems to drop noticeably starting around section 3.2.
- No system figure, no illustration of the contribution.
- What are we supposed to learn from this paper, what questions does it answer?

In particular, I had a hard time understanding the "virtual camera transformation" part. It seemed related to the separation of the moving objects, but it was not clear to me otherwise.
summary_of_the_review: The paper lacks technical novelty, despite the increased complexity (see weaknesses). If I understand correctly, the main contribution of the paper is showing that novel view synthesis of static and dynamic objects can benefit from depth densification and multi-camera reprojection. In my view, the impact of this finding alone is below threshold for publication at ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method for novel view rendering using deep learning based on NERF model where the background and the foreground are synthesized jointly. The NERF model is given extra supervision via LIDAR depths and different confidence maps based on LIDAR depth. The confidence maps help in incorporating standard street view datasets which come with noisy LIDAR depths. A number of existing work is used as building blocks in the paper.
strength_and_weaknesses: Strength:
* The results are good and show that their method works well.

Weakness:
* Sec 3.3: One can use the 3D....the ego car. How is the relative calibration computed. Some more details would be good for clarity.
* Eq.4 is not correct. The correct equation should be (Pi * Pb^-1)^-1 = Pb * Pi^-1
* Sec 3.4 requires some figures and images to clearly explain the different confidence measures and how they are computed.
* In "LiDAR depth completion", when the LIDAR depths are accumulated, they are from different time instants. So, how are they combined and brought to the same coordinate system. For static scenes, the extrinsics between the car can be used to fuse the depths, but for moving objects in the scene like cars this will not work and they will always end up as outlier depths. This whole paragraph is not clearly explained as it appears to be one of the contributions of this paper.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, although the flow of the paper could be improved. The different blocks of their system pipeline are explained but it is challenging to relate the different blocks. This makes reading the paper a bit difficult requiring going back and forth to understand the blocks. Also, the number of figures is very limited to understand the various equations. The authors should have put more effort in putting more figures to geometrically explain their error functions. The paper also uses a number of existing deep learning work e.g. densifying sparse point cloud, camera pose improvement to name a few as their building blocks. Their is no clarity of explanation on what happens when these blocks do not perform well. From the results, it appears that the results are better than existing methods for street view scenes. This may imply that when the system works, it works very good but would have helped to have some analysis of cases where it failed and why. So, reproducibility may be a challenging factor with this paper. 
summary_of_the_review: In summary, the paper has some novel contributions in the way they incorporate LIDAR data and confidence measures which enable use of existing standard datasets for training. The writing could have been improved as the flow of the paper is not smooth. More figures should have been put for better explanation of the different blocks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of neural radiance fields for street views. Considering the low overlap among street view images and the unbounded nature of street scenes, this paper improves the scene coordinate parameterization of existing work and proposes a new method for using sparse and noisy lidar depth as supervision. Experimental results show the performance of the improved method is significantly superior to previous methods. 
strength_and_weaknesses: Strength:

+ Although the idea of using sparse and noisy Lidar depth as supervision is not new in the community, the proposed method of using confidence scores and the learnable weights to balance the different items to generate confidence scores is interesting and smart. 

+ Experimental results seem to be very promising. 

Weakness:

The overall contribution of this paper is a little insignificant. For example, 
(1) one of the claimed contributions, " improve the scene parameterization", is just a small modification on the scene coordinates representation strategy, shown in Eq. (2); 
(2) The "improved camera poses" illustrated in Sec 3.3 are just some engineering processing tricks. 

One of the unique challenges in the street-view neural radiance field is that it contains static background and dynamic objects (vehicles, pedestrians). I was expecting a novel neural radiance field method that could distinguish the two kinds of scenes automatically. However, this paper adopts an existing method to first detect the dynamic object (vehicle) and estimates its pose, and then apply a NeRF-type method to generate the neural radiance field. This is less attractive. 

Other comments:
(1) The paper below also addresses the neural radiance field of unbounded scenes. It may be worth a discussion in the related work. 

Li et al. Neural plenoptic sampling: capturing light field from imaginary eyes. 2021

(2) The second row in Sec 4.5, "out"--> "our"?
clarity,_quality,_novelty_and_reproducibility: The wring of this paper is good and easy to follow. This paper clearly states which parts are from other papers and which parts are proposed by themselves. The originality of the work is good. 
summary_of_the_review: Although I have concerns about the insignificance of some of the contributions, the idea of how to use sparse and noisy lidar points as supervision is interesting and should benefit related tasks where sparse and noisy lidar points are available.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the novel view synthesis of street view and contributes a new NeRF design (named S-NeRF). Since the conventional nerf solutions could easily fail in street view scenes, the authors use a new scene parameterization function and camera poses for learning better neural representations from street views. The experiments demonstrate the proposed method outperforms state-of-the-art works on general self-driving datasets.  
strength_and_weaknesses: + Strengths： 
● The paper is well-written and easy to follow. 
● The depth supervision design is novel and technically sound. The ablation study verifies its contribution as well. More importantly, such a strategy allows us to use low-cost spare Lidar data instead of the dense one in urban-nerf. 
● The experiments show the proposed method outperforms SOTA works in both foreground vehicles and background scenes NVS. 

- Weaknesses： 
● The virtual digital camera transformation process appreciably depends on the quality of 3D detection results. This possible threat should lead to an inconsistent coordinate system transformation.  
● Ablation study:  Based on the ablation study in tab 3 and fig 6, the depth confidence surely helps the proposed model. However, the mechanism behind L_depth is still blurry. It is valuable and theoretically interesting to conduct a more detailed ablation study on confidences of rgb, ssim, vgg, depth and flow and show what technology is critical to this work.
● Discussion about deeper principles: It is not clear why the proposed method gives significantly better results than other methods for background composition using only RGB images. We also need more analysis on the limitations of the proposed method. It would be good if the authors provide some failure case  evaluation to better understand the limitation of the proposed method.
● Make the streets more real: Dynamic objects such as pedestrians, bicycles, and trucks are very common on the streets. Although it is too demanding to reconstruct dynamic objects, it is interesting to study the effect of these dynamic objects on the reconstruction of the target.
clarity,_quality,_novelty_and_reproducibility: Building on existing approaches, the framework proposed in this paper addresses a research question with an efficient approach. This paper draws on so many existing techniques such as all kinds of NeRF, 3-D object detection and common depth estimation, that it's more like a high-level combination. So I think the innovative newness of this paper still needs to be strengthened and can be followed up with more in-depth work.
summary_of_the_review: Overall, this paper presents a novel Nerf framework for self-driving applications. But I still think the author could have done more in-depth research and tweaking.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers raised concerns regarding the technical contributions e.g. that the proposed approach significantly builds on prior work with minor modifications, and side-steps the key challenges of dealing with dynamic objects using off-the-shelf (supervised) 6D pose estimation.  However, on the positive side, the reviewers appreciate the setting studied and the impressive results in the outdoor driving setup. The paper also makes some interesting contributions in how to robustly incorporate LiDAR observations for training NeRFs and presents a useful system with good results in reconstructing neural fields in outdoor driving scenarios. After internal discussion, the reviewers all leaned towards acceptance as this work and the AC concurs.
justification_for_why_not_higher_score: The approach here is a combination of multiple tricks and may not offer any interesting insights/techniques that maybe useful generally beyond the specific setting studied.
justification_for_why_not_lower_score: This work could be is empirically solid and builds a useful system that outperforms prior neural 3D reconstruction approaches in outdoor driving scenes. After the discussion phase, all reviewers recommend acceptance.
Decision: Accept: poster