Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a self-supervised approach for the task of surround-view depth estimation. 2D image features are un-projected onto a shared 3D volume, which is later queried for decoding depth maps at target views. Similarly, pose changes of the canonical frame is also decoded from volumetric feature pooled from all 2D views. Evaluations conducted on DDAD and nuScenes show competitve results compared to FSM and others. 
strengths_and_weaknesses: __Strengths__

Because of the use of a volumetric feature representation, features from images are aggregated in a shared space. This should help produce more consistent depth maps across views. 

Also because of the use of a volumetric feature representation, the proposed model has the unique capability of predicting depth maps for views not among the inputs. 


__Weaknesses__

A few relevant works are not included or discussed sufficiently. The way 2D features are un-projected onto a shared 3D volume aggregation has been used in many places, e.g. [1], [2], [3]. 

[1] Atlas: End-to-End 3D Scene Reconstruction from Posed Images, ECCV 2020. 
[2] Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D, ECCV 2020.
[3] Learning a Multi-View Stereo Machine, NIPS 2017. 

In particular, [3] is very similar as it also fuses 2D features from multiple views on a shared 3D volume which is later projected to 2D views for depth estimation. 

The quality advantage over existing methods is not convincing:
* Although slightly better than FSM quantitatively in some case, the gap is usualy quite minimal and sometimes non-existence. 
* Qualitative comparisons (e.g. in Fig.6) are not showing any obvious improvements. 
* How do the depth maps look if viewed as point cloud? This can also provide insights if the proposed method has a better agreement between adjacent views. 

Because of the volumetric feature representation, there is a trade-off between resolution and compute. This is not a concern for image-space models. 

A few more minor issues:
* The "outer product" operation in Fig.3 is not mentioned anywhere in the text and contradicts the architecture given in the supplementary. 
* The loss terms are not given precise definitions. 
* L.257 ("provide our attempts to reproduce and discussions in the supplementary material.") -- not found in supplementary. 
questions: It's important to address the closely related prior work mentioned above and highlight any new contributions. 

Also, I feel the paper needs to have a more convincing analysis of the differences compared to FSM and others, e.g. examples highlighting typical qualitatively improvements. 
limitations: Adequately addressed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel surround-view depth estimation, and project features of different views into a volumetric feature spaces, where overlapped views can enhance each other and get better depth maps. The pipeline also enables novel-view depth synthesis. The problem setting may be useful for autonomous driving. 

The experiments are insufficient. 
strengths_and_weaknesses: + Paper is easy to follow. 
+ The problem setting is novel. 

- The depth map quality is on par with monocular methods. 
- No visualizations of different methods on DDAD and nuScenes are provided. From Figure 5, the depth maps are similar to existing monocular methods in quality. 
- In Table 2, depth fusion is not evaluated individually. 


questions: - Is it possible for the proposed method to compare with monocular methods? There are many common benchmarks for monocular depths. 
- Would the cubic space introduce some artifacts and distortions comparing to spherical spaces? 
- As a self-supervised method, I assume there is no GT depth needed? Is that so? Is it possible to train on real data without GT? 
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a self-supervised depth estimation method based on a unified volumetric feature representation encoded from surround-view. The proposed method consists of three parts. First, the surround-view feature fusion module generates a unified volumetric feature from the extracted multi-view 2D image features. Second, the depth fusion module reconstructs the depth map given an input camera viewpoint. Last, the global motion of the canonical camera is estimated by assuming static camera extrinsic. Experiments on DDAD and nuSecens datasets achieved improved results over existing methods.

strengths_and_weaknesses: Strengths:
- The idea of using a volumetric feature to encode information from surround-view images seems interesting and logical.
- With the construction of volumetric feature and depth fusion, the proposed method achieves improved results on DDAD and nuScenes datasets.
- Ablation study has been provided to verify the effectiveness of depth fusion and canonical motion estimation.


Weaknesses:
- The core of this method is the construction of the unified volumetric feature to estimate the surround-view depth and canonical camera motion. However, some details for constructing a unified volumetric feature are missing. This method simply copies the image feature to all voxels went through by the corresponding pixel ray. In this case, some voxels may have been passed by multiple pixel rays. It is unclear how to handle multi-feature collisions.
- Related to the first question, why different MLPs are used to fuse the overlap features and non-overlap features in the volumetric feature (see Line 132)? No experiments are shown to verify this strategy.
- Since this method encodes the feature in a 3D volume, the effect of volume resolution on the memory consumption and depth estimation should be discussed.

Overall I think the proposed idea is interesting, but I have some questions for the volumetric feature construction part. I would give a borderline accept rating at this stage.

questions: - More details for the construction of the volumetric feature.
- Effect of volume resolution on memory consumption and depth estimation.
limitations: The authors have discussed the limitation and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: Initially, the paper had mixed reviews (455).  The major concerns from the reviews were:

1. missing refs about unprojection. (K314)
2. quality advantage is not convincing, slightly better than FSM, while qualitative results show not obvious improvements. (K314)
3. visualize the depth maps as point clouds (K314)
4. what is the trade-off between resolution/memory, computation, and depth estimation? (K314, ZBww)
5. insufficient experiments (9Ef1)
6. comparison with monocular methods (9Ef1)
7. cubic vs spherical space? (9Ef1)
8. can it be trained on real data w/o GT supervision? (9Ef1)
9. how to handle collision of multiple pixel rays? (ZBww)
10. why use different MLPs to fuse "overlap" and "non-overlap" features? No ablation study on this. (ZBww)

The authors wrote a response to address these concerns, providing more qualitative results and ablation studies, as well as further explanations.  The reviewers were satisfied with the response, and K314 upgraded their rating to 6, while other reviewers maintained 5s. The reviewers appreciated the novel problem and the solution that can produce more consistent depth maps across views, and also synthesize depth maps in novel views. After reading the paper, the AC agrees with the reviewers, noting that the paper  addresses the limitations of the problem setup of previous work [13], thus developing a new line of research.  Thus, the AC recommends accept. The authors should prepare a revised version of the paper according to the reviews, rebuttal, and discussion. 
confidence: Certain
Decision: Accept