Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper aims at a unique problem of multi-camera surrounding view self-supervised depth estimation. A cross-view transformer is designed and applied at multiple scales to incorporate multi-view features. Comprehensive experiments are conducted for depth evaluation.

strengths_and_weaknesses: Strengths:
+ This research is well-motivated for multi-view cameras on autonomous vehicles.
+ This paper is generally well-written in a good structure.
+ This paper mostly achieves SOTA performance in DDAD and nuScenes datasets.

Weaknesses:
- The proposed method cannot theoretically guarantee the multi-view depth consistency.
- The computational performance is not evaluated for the robotic context.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper presents a novel cross-view transformer to fuse multi-camera features in a multi-scale fashion for a unique problem - multi-surrounding view camera depth estimation. The application is novel, although the methodology is still considered incremental. Given the performance in this unique problem, the review suggests 'Weak Accept' or slightly below that.

issues: - The scientific novelty and method generality should be highlighted. Since the method cannot theoretically guarantee the multi-view depth consistency, the authors are suggested to construct a few research hypotheses for the scientific novelties.
- Joint pose estimation is included but not fully evaluated (preferred).
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposed a method that predicts high-quality depth maps across cameras by exploring correlations among surrounding views. They apply the cross-view attention to fuse the downsampled feature maps. They use structure-from-motion on adjacent frames to obtain the scale-aware pseudo depths. To exploit the pose consistency between views, they propose to predict the universal pose and then transform it to each local camera pose instead of predicting the pose of each view separately.

strengths_and_weaknesses: Strength
This paper is well written and easy to follow. Their method shows good results on two datasets compared to the existing works. The experimental results show that they could effectively incorporate the information from multiple views. They provide insights to supervise the depth network to learn the real-world scale. The joint pose estimation proved to be having a positive effect on the performance. Also, the reported model’s inference time is fast.

Weakness
The reported scores of tables 1-4 are averaged over all views. Are the performance gains consistent on each separate view?
As shown in table 4, the setting of w/o spatial context might lead to worse results on the metrics of Sq Rel. Also, the performance gap between w/o spatial context and the whole setting is smaller than that on the DDAD dataset. The area of overlaps between adjacent views might be different in the different datasets. Does it bring some restrictions to this method?
FSM uses a more light-weight encoder(Resnet-18) compared to the ResNet-34 using here. It is good to see the information on the FSM’s inference time.
This work proposes to exploit information from multiple views to make predictions. In real applications, it is common to have extrinsic noises, one camera occluded, or camera time delay. It is good to see the robust analysis.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I think results and the techinical solution are convincing and interesting. I would recommend weak accept.
issues: Please address my concerns in the weaknesses.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a novel method for multi-view depth prediction based on multiple color images without the usage of calibrated depth sensor (LIDAR). They propose extracting features with a simple backbone (ResNet34), which are combined with a Cross-View-Self-Attention method, where the final results are deconvoluted and combined with skip connections from the encoder to keep detail.
They show how to predict a scale for the depth estimations based on a nice pretraining on structure-from-motion data.
Furthermore, they predict the joint pose for all cameras, showing that they even can outmatch the error in the ground truth data.
They also show superior results in comparison to FSM, PackNet-SfM and Monodepth2.

strengths_and_weaknesses: Strengths:
* The paper is well written and easy to follow, something I do not say lightly
* I especially like the idea of using an attention model for incorporating the information from the different views
* great ablation, most questions I had were evaluated and proven to be improved

Weaknesses:
* the only thing I do not fully see proven, but it might be that I need a minor clarification, is that the method works better when it uses all images simultaneously. I feel these results are covered in table 8., but they are not fully discussed in 4.3. Can I get a clarification on this.
* one thing I particularly do not like is when depth images are solely represented as images and not as point clouds. It is nearly impossible to judge a depth image in 2D, anyone who claims they can judge that is lying. I would have love an image or even better, a video of a point cloud, with the different metrics turned on and off to immediately see the difference
 
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I strongly recommend this paper for acceptance to CORL. They have proven that this paper deserves to be published in CORL. 



Old Review before Discussion: I will be absolutely clear in my review. I think this paper deserves a strong accept. The idea is good the evaluation is great. I just have one question, can you prove that using multiple images with the cross-view-self-attention method improves the results? If that is already hidden in table 8. please tell me, and I will gladly change my review to strong accept. If not, please add these results, and I will also change my review.
issues: 
Not relevant anymore: 
* more clearly show that your multi-view approach improves over using a non-multiview approach
* add a point cloud visualization to the supplementary
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposed a SurroundDepth module to predict the full surround depth maps from multiple cameras. 
The major contribution includes: 
* A novel cross-view transformer with cross-view self-attention to fuse the multi-view information and enable interactions. 
* An SfM(structure-from-motion) pretraining mechanism is proposed to predict the metric scales of the depth.
strengths_and_weaknesses: **Strengths:**
* Although the full-surround depth prediction is not a new concept(e.g. FSM[29]), the author proposed a novel cross-view transformer and self-attention mechanism to better fuse the feature and enable interactions to improve the accuracy.
* The proposed Scale-aware SfM pretraining is effective to retrieve the metric scale for depth prediction.
**Weaknesses:**
* The experiment results in Table3 and 4 are inconsistent with paper FSM[29], it is better to use the same evaluation settings or at least illustrate the detailed difference in evaluation.
* It's better to also test on the KITTI dataset with the Stereo setting to better compare with the majority of the depth prediction methods.
* The comparison of test inference time with FSM[29] is missing, what is its inference time on the same hardware setting?
* It's better to also compare the number of parameters, consumption of GPU-RAM, and training time with the baseline methods. 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The author introduced the Transformer into full-surround depth prediction task and achieved significant performance improvement according to the experiment section. However, the evaluation settings need to be further illustrated and the computational cost should be further analyzed.

Post-rebuttal: 
(1)The evaluation comparison with FSM[29] should be updated. 
(2)The proposed method consumes significantly more computation and memory while the performance improvement looks marginal.
(3)The introduction of Transformers to this task and the open-source code do contribute to the community, which is appreciated.
Based on the above considerations, my final rating is "weakly accept"
issues: **Major Issues:**
* Since the FSM[29] is the only fair comparison baseline, could you use the same evaluation settings or at least illustrate the detailed difference in evaluation?
* The Transformers are generally computational heavy, could you further compare the computational cost such as the number of parameters, consumption of GPU-RAM, and training time.
**Minor Issues:**
* The KITTI Monocular/Stereo performance results are appreciated if possible.
* The limitation analysis could be more comprehensive.
review_title: null

Meta-review: metareview: The paper received overall favorable reviews pre-rebuttal. The reviewers agree that the proposed method is novel, technically sounds, and performant. They also acknowledge the presentation quality. To improve the manuscript, the reviewers suggest that the author should (1) clarify assumptions regarding view overlaps (RZGD), (2) robust analysis (RZGD), (3) further clarify if using multiple views clearly improve the performance (maRz), (4) visualize result in point cloud (maRz), (5) make evaluation setting consistent with that of FSM (nGH3), and (6) discuss/evaluate the computation performance of the method in practical settings.

Post rebuttal: The authors have addressed majority of the concerns during the response period. The reviewers have all agreed that the paper studies an important problem and provides a convincing solution, provided that the paper will be updated with the following content: (1) comparison with FSM, (2) justification of multi-view performance gain, (3) open source code, and (4) detailed analysis of runtime performance. The paper is recommended to be accepted and presented as a poster. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)