SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation
Authors: Yi Wei
Linqing Zhao
Wenzhao Zheng
Zheng Zhu
Yongming Rao
Guan Huang
Jiwen Lu
Jie Zhou
Venue: CoRL 2022 Poster (2022)
Keywords: Self-supervised depth estimation
Multi-camera perception
Structure-from-motion

Abstract: Depth estimation from images serves as the fundamental step of 3D perception for autonomous driving and is an economical alternative to expensive depth sensors like LiDAR. The temporal photometric consistency enables self-supervised depth estimation without labels, further facilitating its application. However, most existing methods predict the depth solely based on each monocular image and ignore the correlations among multiple surrounding cameras, which are typically available for modern self-driving vehicles. In this paper, we propose a SurroundDepth method to incorporate the information from multiple surrounding views to predict depth maps across cameras. Specifically, we employ a joint network to process all the surrounding views and propose a cross-view transformer to effectively fuse the information from multiple views. We apply cross-view self-attention to efficiently enable the global interactions between multi-camera feature maps. Different from self-supervised monocular depth estimation, we are able to predict real-world scales given multi-camera extrinsic matrices. To achieve this goal, we adopt two-frame structure-from-motion to extract scale-aware pseudo depths to pretrain the models. Further, instead of predicting the ego-motion of each individual camera, we estimate a universal ego-motion of the vehicle and transfer it to each view to achieve multi-view consistency. In experiments, our method achieves the state-of-the-art performance on the challenging multi-camera depth estimation datasets DDAD and nuScenes. 

SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation
Yi Weil!2; Linqing Zhao"; Wenzhao Zheng 12, Zheng Zhu Yongming Raol.2
Guan Huang 4, Jiwen Lu Lu Jie Zhou
Beijing National Research Center for Information Science and Technology, China *Departeent Automation, Tsinghua University, China School of Electrical and Information Engineering, Tianjin University, China *PhiGent Robotics
Abstract: Depth estimation from images serves as the fundamental step of 3D perception for autonomous driving and economical alternative expen- sive depth sensors like LiDAR. The temporal photometric constraints enables self-supervised depth estimation without labels, further facilitating its application. However, most existing methods predict the depth solely based on each monocular image and ignore the correlations among multiple surrounding cameras, which are typically available for modern self-driving vehicles. In this paper, we propose a Derh method incorporate the information from multiple surrounding views to predict depth maps across cameras. Specifically, we employ joint net- work to process all the surrounding views and propose a cross-view transformer to effectively fuse the information from multiple views. We apply cross-view self- attention to efficiently enable the global interactions between multi-camera feature maps. Different from sulf--upervised monocular depth estimation, predict real-world scales given multi-camera extrinsic matrices. To achieve this goal, we adopt the two-frame Atructure-fromt taaeee-froommiiooo motion to extract scale-aware pseudo depths to pretrain the models. Further, instead of predicting the ego-motion each individual camera, estimate universal ego-motion of the vehicle and transfer it to each view to achieve multi-view ego-motion consistency. In exper- iments, our method achieves the state-of-the-ar performance on the challenging multi-camera depth estimation datasets DDAD and nuScenes. Code is available https:///wibbbbccom///vvtt/u/ur/
able
Keywords: Self-supervised depth estimation, Multi-camera perception, Structureffrommmotiio
Introduction
Recent years have witnessed the rapid development of autonomous driving. Instead of relying on expensive depth sensors like LiDAR to perform extract structural information, 3D perception from cameras has become promising approach and potential alternative due its semantic richness and economy. Acting as bridge between the input 2D image and the real 3D environment, depth estimation has crucial influence on the downstream 3D understanding and receives increasing attention [1, 2, 3, 4, 5, 6, 7, 7,8,9 9, 10].
Due to the expensive cost of densely annotated depth maps, depth estimation is usually learned in self-supervised manner. By simultaneously predicting the depth maps and the ego-motions of cam- eras, existing methods take advantage of the temporal photometric constraints between successive images as the supervision signal [11, 12, 13, 14, 15, 16). Despite that modern self-driving cars
"Equal contribution.
t Corresponding author.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

SurroundDepth: Entangling Surround Views
Conventional: Individual
A
Self- Supervised Depti Estimation
Comparison between Derth and self-supervised monocular depth estimation methods. Conventional methods [27, 28] predict depths and gro-motions of each view separately, which ignores the correlations between views. Our method incorporates the information across cameras and jointly process all surrounding views.
Figure 1:
are usually equipped with multiple cameras to capture the full 360" view of the surrounding scene, most existing methods still focus on predicting depth maps from monocular images and ignore the correlations [17, 18, 19, 20, 21, 22, 23, 24, 25, 26] among the surrounding views [27, 28, 29, 30, 31].
In this paper. we propose SurroundDepth high-quality depth maps across cameras. We first employ a shared encoder to extract high-level feature maps for each view and then propose cross-view transformer to effectively fuse them. To efficiently allow multi-scale interactions between the features across cameras, we first reduce the resolution of feature maps with depthwise separable convolutions. Then we apply cross-view tention to integrate multi-camera features and use deconvolution to recover the original resolution. To alleviate the information loss induced by feature map downsampling, we add skip connections between individual and interacted feature maps. Finally, we use a shared decoder to attain the predicted depth maps. To recover the real-world scales, we propose Structure cenetture-foo--eee Prom-Motion (SfM) pretraining and joint pose estimation to predict scale-aware results. In detail, we pretrain the depth estimation model with scale-aware pseudo depths derived from SiM [32]. Joint pose estimation is designed to achieve consistent pose predictions between cameras. By explicitly leveraging ex- trinsic matrices, we are able to calculate the individual ego-motion of each view from the predicted universal ego-motion. We conduct extensive experiments on the challenging multi- multi-camera depth es- timation datasets DDAD [27] and nuScenes [33]. Experimental results show that the cross-vine view feature interaction booost the performance and the proposed techniques can achieve real-scale depth estimation.
process all the surrounding views jointly to produce
2 Related Work
Self-supervised Monocular Depth Estimation: Without available ground-truths, many approaches explore the routes of learning depths and motions simultancously [11, 28, 12, 13, 14, 34, 15, 16, 35]. For monocular sequences, the geometric constraints are usually built on adjacent frames. Zhou er al. [11] built the problem as task of view synthesis and trained two networks to separately pre- dict poses and depths. Monodepth2 [28] further enhanced the quality of predictions by proposing the minimum re-projection loss, full-resolution multi-scale sampling, and auto-masking loss, which has been adopted [36, 37J. Recently, FSM [29] extended self-supervised monocular depth es- timation to full surrounding views by introducing both spatial and temporal contexts to enrich the supervision signals. Different from these monocular depth estimation methods, Our Surroundeepth captures cross-view interactions in surrounding views, which are important for fully understanding the environment with multiple cameras.
Additional Supervision for Depth Estimation: Recent approaches introduce additional supervi- sion signals to strengthen the accuracy of depth estimation, such as optical flow [14, 38] and object motion [36, 16]. DispNet [39] was the first work to transfer information from synthetic stereo video datasets to the real-word depth estimation. Besides, Zheng el al.[40] proposed a two-module do- main adaptive network with generative adversarial loss to transfer knowledge from the synthetic domain. Some methods adopt auxiliary depth sensors to capture accurate depths, such as LiDAR, to assist depth estimation [41, 42]. To predict the depths with real-world scales, our method lever-

---

target
CVT
CVT
CVT
mult-cammer images NoHooW33
depth prediction NxHadW
protrain
front left
front
front right
two-frame Structurefrom--Motion
scale-aware pseudo depths
P!
R
Pr-us
Pt (T*) Pt-sT"
network
ego-motion
photometric consistency
source frames
An overview of our SurroundDepth. We utilize encoder-decoder networks to predict depths. To entangle surrounding views, propose the cross-view transformer (CVT) fuse multi- camera features in multi-scale fashion. Pretrained with the sparse pseudo depths generated by two-frame Pruucturefrom-Mootion the depth model is able learn the absolute scale of the real world. By explicitly introducing extrinsic matrices into pose estimation, we can predict multi-view consistent ego-motions and boost the performance scale-aware depth estimation.
Figure 2:
ages multi-camera extrinsic matrices and takes the sparse depths obtained from StM to pretrain our network.
3 Approach
3.1 Problem Formulation
In the self-supervised depth and ego-motion settings, the depth network F and pose network G are optimized by minimizing per-pixel photometric reprojection loss [43, 11]. This reconstruction- based self-supervision paradigm achieves great progress in monocular depth estimation methods [1 1, 12], and can be directly extended into multi-camera full surround depth estimation. The predicted depth maps and poses of
of input surrounding samples {I' 12, INN can be written as:
D. = (D.' (F P.., = (Pi IN = (G(T,T))
(1)
where subscript s and mean the source and target frames. However, estimating full surrounding depths differs from monocular depth estimation in that there exists crucial correlations among the surrounding views, as shown Figure The overlaps between adjacent views connect all views into full 360 view of the surroundings, which contains lots of beneficial knowledge and priors for

---

understanding the whole scene. Based on this fact, we build the joint models to predict depths and ego-motions instead of view-dependent estimations:
D. = (D:) = P (P% -(U (D)))
Profiting from the joint model, we can not only enable the cross-view information interactions to improve the performance all views but also generate the universal ego-motion to produce scale- aware predictions with camera extrinsic matrices.
3.2 Overview
As illustrated in Figure 2, the network can be separated into three parts (i.e., a shared encoder E, shared decoder D, and several cross-view transformers (CVT)). Given set of surrounding images, the encoder networks first extract their multi-scale representations in parallel. Different from existing methods that directly decode the learned features, we entangle the features from all views into an integrated feature at each scale, and further utilize multiple scale-specific CVT to perform cross-view self-attentions over all scales. By taking advantage of the powerful attention mechanism, CVT enables each element on the feature maps to propagate its information to other positions and absorb information from others at the same time. Finally, we separate the post-interactive features back to N views and send them to the decoder D.
Unlike monocular depth estimation, we are able to recover the real-world scale from camera extrinsic matrices. A straightforward method to leverage these camera extrinsic matrices is to embed them into spatial photometric loss between two adjacent views. However, we find that the depth network fails to learn the scale directly supervised by spatial photometric loss. To tackle this issue, we propose scale-aware SIM pretraining and joint pose estimation. Specifically, we use two-frame SIM to generate pseudo depths to pretrain the models. The pretrained depth network able to learn the real-world scale. Moreover, the temporal ego-motions of N cameras have explicit geometric constraints. Instead of using a consistency loss, we estimate the universal pose of the vehicle and calculate the ego-motion of each view according to their extrinsic matrices.
3.3 Cross- Vress-View Transformer
With the multi-scale features extracted from all surrounding views, we replace the skip connections between the encoder and decoder with our proposed cross-view transformer (CVT), as shown in Figure 3. Let X3 E IR N H, x X.x be the feature maps obtained from the k-th scale, and we first supplement each feature with three unique learnable positional encodings (PEx,PE,,PEw which represent the view-wise, row-wise and column- wiie index, respectively.
Then we build 7 cross-view self- afff-ateention layers to perform the cross view information exchanging, whose computation cost is O HH&W? for each layer. However, the cost would be too enormous to accept when processing larger feature maps from the lower layers of the encoder. To avoid the huge cost caused by the quadratic term (i.e., H W we place a depthwise separable convolution (DS- Conv) [44, 45] before attention layers first summarize the large feature maps into lower-resolution ones with same channel numbers, i.e., X' E R xh, Xws Xd. With much less computation than a standard convolution, the DS-Conv is widely adopted to balance the trade-off between efficiency and performance. To create the pathway across the features of surrounding views, we flatten the feature maps into unified sequence, which includes the elements from all views, i.e., N x h4 x wk elements in total. We develop three linear layers to obtain the query, key, and value vectors from X' + PFN PF PFw. and then split the features into multiple groups along the channel dimension for multi-head self-attention to enrich the feature diversity. The output features can be represented as:
O Sofumax KT Qi/Vdd)V
(3)
aut = Concat(Orutt (O.ta......
where M is the number of feature groups, and K,, Q., V; denote the i-th feature group of the key, query, and value features, respectively. Benefiting from this attention unit, we ensure that every element on X's not only interacts with intra-view features for better understanding of the current frame but also receives inter-view contexts for comprehensive cognition of the scene. To make

---

Skip Connection
Cross-View Self-Attention
DS Conv
DS
No KOK
Nehi
NKH
NHH
Figure 3: The proposed cross-view transformer. We first use a depthwise separable convolution (DS-Conv) layer to summarize the multi-view features into compact representations. Then we build cross-vies view self-attention layers fully exchange the flattened multi-view features. After the cross-view interactions, we use a DS-Deconv layer to restore the resolutions of multi-view features. At last, we construct skip connection combine the input and restored multi-view features.
cross-view information fully interactive, stack such unit for Z times in cascade style. As an inverse process of flattening, we reshape the outputs of the Z-th attention layers back to the shape of N hu X We X d4, and further upsample the features to the original resolution of input features through a depthwise separable deconvolution. To alleviate the information loss caused by downsampling the feature maps and preserve the input details, we construct skip connection to directly combine the input and output features:
D Xx X
(4)
By extending this strategy, features of all scales and all views can be fully propagated and updated from a global perspective.
3.4 Scale-aware Structure-from-Motion Pretraining
The aim of SfM pretraining is to explore the real-world scale from camera extrinsic matrices. A direct way to leverage extrinsic matrices is to spatial photometric loss between two neighboring views, i.e., warping I to I/::
P - - -"==""())--''<(((>>>
(5)
where Kt, T are the intrinsic and extrinsic matrices of ith camera. However, the overlap between I' and 1s is relatively small, and the p, + is easy to be out of the image bounds if the scale of D far from the real-world scale. In this way, the start of the training, the spatial photometric loss will be invalid and cannot supervise the depth network to learn the scale.
To tackle the problem, first adopt SIFT [46] descriptors to extract correspondences. Then compute scale-aware pseudo depths by triangulation with camera extrinsic matrices. Finally, we leverage these sparse pseudo depths along with the temporal photometric loss pretrain the depth and pose networks. However, due to the small overlap and large view changes, the robustness and accuracy of descriptors decrease. To alleviate the issue, we only find corresponding points in a certain region instead of the whole image since we can roughly know the overlapping regions between two neighboring views. Specifically, in this work, we define left part of I' and right part of / r++1 as the valid regions. Further, we leverage epipolar geometry to filter outliers. The epipolar line of point 9'n can be described as:
Im = Fi-ez4m
Pi,,- =
(6)
[R| (77)-171
where Pi-.y is the fundamental matrix. If the distance between o/ and epipolar line Im is larger than threshold 7, we regard this correspondence pair as the noise. We provide an example in the supplementary materials.
5

---

Method
Abs Rel
Sq Rel
RMSE
RMSE log
1.25
8 < 1.25*
Ó 1.25
Monodepth2 [28]
0.362
14.404
14.178
0.412
0.683
0.859
0.922
PackNet-SIM [27]
0.301
5.339
14.115
0.395
0.624
0.828
0.908
Monodepth2 -M
0.217
3,641
12.962
0.323
0.699
0.877
0,939
PackNet-StNl -M
0.234
3.802
13,253
0.331
0.672
0.860
0,931
FSM [29]
0.202
FSM [29]
0.229
4.589
13.520
0.327
0.677
0.867
0.936
SurroundDepth
0.200
3.392
12.270
0.301
0.740
0.894
0.947
Table 1: Comparisons for self- sepersupe multi-camera depth estimation on DDAD dataset [27]. All methods are trained and tested with the same experimental settings. The results are averaged over all views with median-scaling at test time. -M indicates occlusion masking. indicates our implementation.
Method
Abs Rel
Sq Rel
RMSE
RMSE log
< 1,25
ò < 1.25*
5 < 1.25
Monodepth2 [28]
0.287
3.349
7.184
0.345
0.641
0.845
0.925
PackNet-SIM [27]
0.309
2.891
7.994
0,390
0.547
0.796
0.899
FSM [29]
0.299
FSM [29]
0.334
2.845
7.786
0,406
0.508
0.761
0.894
SurroundDepth
0.245
3.067
6,835
0.321
0.719
0.878
0.935
Table 2: Comparisons for self-supervised multi-camera depth estimation the nuScenes dataset [33]. All methods are trained and tested with the same experimental settings. The results are aver- aged over all views with median-scaling at test time. indicates our implementation.
3.5 Joint Pose Estimation
An intuitive way of pose estimation is to predict the relative pose of each view separately, which can be represented by Equation 1. However, this strategy ignores the pose consistency between different views, which may lead to ineffective supervision signals. To maintain multi-view ego- motion consistency, we propose to decompose the camera pose estimation problem into two sub- problems: universal pose prediction and universal-to-local transformation. To obtain the universal pose P, we feed N pairs of target and source images into the PoseNet G' at time and take the average of extracted features before the decoder. The universal pose can be computed by:
{ (TE((((T,,T)))) N
Pt P
Σ h())
N
1=1
where Ge and Gp denote the encoder and decoder of PoseNet, respectively. After obtaining the universal pose P, we can further transform it to each camera pose with known camera extrinsic matrices, which can be described as:
P P
where 7 is the extrinsic matrix of ith camera and P is its ego-motion from target frame to source frame. By combining these two steps, we can obtain the theoretically consistent multi-camera poses, which will further improve the scale-aware depth estimation accuracy.
4 Experiments
4.1 Experimental Setup
We conduct experiments on both the Dense Depth for Automated Driving (DDAD) [27] and nuScenes datasets [33]. The basic backbones of depth and pose networks are the same as the Mon- odepth2 [28]. We employ ResNet34 with ImageNet [47] pretrained weight as the encoder for all experiments, including baseline methods. Further, surrounding cameras have different focal lengths and we refactor depth maps according to focal lengths following [48]. In each scale, we adopted transformer layers and features were downsampled to 20 and 20x 11 before cross- view attention on two datasets respectively. Since DDAD dataset has self-occlusion areas, following [29], we annotate occlusion masks manually and use them reweight photometric loss. More implementation details and visualizations in supplementary materials.

---

spatial context
SiM pretrain
joint pose
Abs Rel
Sq Rel
RMSE
RMSE
8 < 1.25
Ó < 1.25
< 1.25a
log
0.967
22.982
31.761
3.518
0.000
01.00)
0,000
0.978
24.062
31,980
3.749
0.000
0.000
0.000
0.257
4.565
14.096
0.368
0,557
0.833
0,925
0.881
19.499
29.552
2.224
0,000
0.001
0.002
0.411
6.121
17.747
0.626
0.089
0.367
0.767
0.208
3.371
12.977
0.330
0.693
0.871
0.934
Table 3: Quantitative results for scale-aware depth estimation on the DDAD dataset [27]. The scores are averaged over all views without median-scaling at test time. 'spatial context' denotes whether to use spatial context and multi-camera extrinsic matrices for training.
spatial context
StM
joint pose
Abs Rel
Sq Rel
RMSE
RMSE log
5 1.25
8 < 1.25
< 1.253
pretrain
0.978
13,906
18.063
3.967
0.000
0.000
0.000
0.970
13.702
17.931
3.654
0.000
0.000
0,001
0.429
7,839
8,593
0.428
0.471
0.797
0.895
0.969
13.661
17.896
3.620
0,000
0.000
0.001
0.363
3,999
8,499
0.479
0.234
0.726
0.876
0.280
4.401
7.467
0.364
0.661
0.844
0.917
Table 4:
Quantitative results for scale-aware depth estimation on the nuScenes dataset [33]. The scores averaged over all views without median-scaling test time. 'spatial context' denotes whether to use spatial context and multi-camera extrinsic matrices for training.
4.2 The Benchmark for Multi-camera Depth Estimation
We compare our method with two existing state-of-the-art self-supervised monocular depth esti- mation methods (Monodepth2 [28] and PackNet-SfM [27]) and one multi-camera depth estima- tion method FSM [29]. Since FSM does not release code, its detailed experimental setting is un- known, such as evaluation, pretraining and hyperparameters. To fairly compare, we run all meth- ods under the same training and evaluation settings. Specifically, the input images are downsam- pled to x34 and 640x352 resolutions. Depth ground truth is generated by projecting LiDAR point cloud to each view. Since Monodepth2 and PackNet-SIM are not designed for scale-aware depth estimation, we perform per-image median ground truth scaling [11] and post-processing [43] during evaluation. Moreover, SfM pretraining and joint pose estimation aware depth estimation, we do not adopt them in this experiment. As shown Tables and 2, method achieves state-of-the-art performance. Our method outperforms baseline method Mon- odepth2, which demonstrates that our proposed cross-view approach improves the results since we fully entangle surrounding views. We test inference time of batch (6 images) single RTX 3090: Monodepth2 (0.028s), PackNet-SIM (0.471s), SurroundDepth (0.088s).
designed for scale-
4.3 Scale-aware Self-supervieed Depth Estimation
Different from monocular depth estimation, once we get the extrinsic matrices of each camera and use spatial context, the absolute scale of the world can be obtained. Tables 3 and 4 show the results without -ecaiin- at test time. We do not compare other methods since these methods can- not predict scale-aware depth maps. In contrast to the statement in [29], we find that the network directly utilizing spatial photometric loss cannot force the network to generate scale-aware depths. Without pretraining, the scale of initial depth is greatly different from the ground truth, which leads to invalid spatial photometric loss. With our proposed SiM pretraining, the depth network learns the absolute scale and spatial context becomes effective. The joint pose estimation further boosts the performance.
We further evaluate the depth consistency between surrounding views. Specifically, we project each view's ' depth map to its neighboring views with extrinsic matrices and calculate depth errors. Table shows that the scale-ambiguous methods such as FSM and Monodepth2 cannot achieve multi-view consistency since the depth maps are scale-ambiguous but extrinsic matrices are scale- With the view interaction, we can further boost the consistency.

---

Pose type
Abs Rel
Sq Rel
0 1.25
skip multi-scale
Abs Rel
Sq Rel
< 1.25
separate
0.257
4.565
0.557
0.217
3.641
0.699
constraint
0.254
4.390
0.576
0.209
3.385
0.704
joint 0,208
3.371
0.693
0.222
3.763
0.695
ground truth
0.210
4.164
0.729
0.200
3.392
0.740
The ablation study for joint pose estima- tion. "separate" indicates that predict the pose of each view separately. 'constraint' further adds multi-camera consistency constraints.
Table 6:
Table 5: The ablation study for cross-view trans- formers. 'skpp' and 'multi-scale' denote skip con- nection and multi-scale cross view transformer.
region epipolar mask filter
Abs Rel
Sq Rel
ő 1.25
scale- ewwree
CVT
Abs Rel
Sq Rel
0 < 1.25
0.241
3,970
0.629
0.989
28.872
0.000
0.236
3.684
0.649
0.319
18.217
0.732
0.208
3.371
0.693
0.257
8.027
0.735
Table 7: The ablation study for SfM pretrain- ing. 'region mask' indicates the region constraint for SIFT matching. "epipolar filter' indicates the epipolar constraints to filter out noisy labels.
Table 8: Depth consistency evaluation.
'scale-
aware' indicates the scale-aware model. 'CVT" means cross-view transformers.
4.4 Ablation Study
In this subsection, we conduct ablation studies to verify the effectiveness of each module our framework. All experiments are conducted on DDAD dataset.
Cross-view Transformer: Table 5 shows that both skip connection and multi- formulation contribute to the final results. Skip connections are able to alleviate the information loss caused by the downsampling of feature maps. The multi-scale cross-view transformer helps the network to better learn both fine-grained and global features.
Joint Pose Estimation: Table 6 shows separate pose estimation cannot guarantee the multi-camera pose consistency. To provide a straightforward baseline (second line), we project the ego-motion of all views the front view and add loss between the projected and predicted ego-motion of the front view. However, this method gets marginal improvements. We note that our joint pose estimation method gets comparable performances with ground-truth pose experiment. This result indicates that ground truth ego-motion may have noise, even for carefully-calibrace public datasets. Thus, it is necessary to predict accurate scale-aware poses.
Scale-aware SEM Pretraining: Table 7 shows that region mask is able to help our model to learn real-world scale. If we do not region mask for SIFT matching, will get wrong correspon- dences and the depth network is not able to learn real-world scale. By filtering out the correspon- dences far from the epipolar lines, we will further boost the performance of the depth network.
5 Limitations
Although the proposed SurroundDepth can boost the multi-view depth consistency with scale-aware training strategy and gross-view transformers (as shown in Table 8), our method cannot theoretically guarantee this consistency. As future work, we will directly predict voxel occupancy of the 3D space instead of each view's 's depth maps.
Conclusion
In this paper, propose SurroundDepth for self-supervised multi-camera depth estimation. The core insight of our method is to entangle multi-camera information and jointly process all surround- ing views. The cross- view transformer is performed at multiple scales to incorporate multi-view features. To obtain scale-aware depth predictions, we present Structure-from Motoon pretraining and joint pose estimation to fully leverage multi-camera extrinsie matrices. Our method achieves state-of-the-a performance on the multi-camera depth estimation datasets.

---

Acknowledgments
This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700802, part by the National Natural Science Foundation China under Grant 62125603 and Grant U1813218, in part by grant from the Beijing Academy of Artificial Intelligence (BAAI).
References
[1] B. Li, C. Shen, Y. Dai, A. Van Den Hengel, and M. He. Depth and surface normal estimation from monocular images using regression deep features and hierarchical crfs. In CVPR, pages 1119-1127, 2015.
[2] P. Wang, X. Shen, Z. Lin, S. Cohen, B. Price. and A. L. Yuille. Towards unified depth and semantic prediction from single image. CVPR, pages 2800-2809, 2015.
[3] A. Roy and S. Todorovic. Monocular depth estimation using neural regression forest. CVPR, pages 5506-5514, 2016.
[4] Liu, C. Shen, G. Lin, and Reid, Learning depth from single monocular images using deep convolutional neural fields. TPAMI, 38(10):2024-2039 2015.
[5] H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao. Deep ordinal regression network for monocular depth estimation. In CVPR, pages 2002-2011, 2018.
[6] M. Song, S. Lim, and W. Kim. Monocular depth estimation using laplacian pyramid-based depth residuals. TCSVT, 2021.
[7] Y. Cao. T. Zhao. K. Xian, C. Shen, Z. Cao, and S. Xu. Monocular depth estimation with augmented ordinal depth relationships. TCSVT, 2020.
[8] J. Huang, G. Huang, 7. Zhu, Y. Yun, and D. Du. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv221/2.11790, 2021.
[9] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Y. Qiao, and Dai. Bevformer: Learning bird's- eye-view representation from multi-camera images via spatiotemporal transformers. arXiv preprint arXiv:2203.17277 2022.
[10] Y. Wang, V. C. Guizilini, T. Zhang, Y. Wang, H. Zhao, and J. Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In CoRL, pages 180-191, 2022.
[11] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe. Unsupervised learning of depth and ego- motion from video. In CVPR, pages 1851-1858, 2017.
[12] J.-W. Bian, Z. Li, N. Wang, H. Zhan, C. Shen, M.-M. Cheng, and Reid. Unsupervised Scale- consistent Depth and Ego-motion Learning from Monocular Video. In NeurIPS, pages 35-45, 2019.
[13] W. Zhao, S. Liu, Y. Shu, and Y.-J Liu. Towards Better Generalization: Joint Depth-Pose Learning without PoseNet, CVPR, 2020.
[14] Z. Yin and J. Shi. Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In CVPR, pages 1983-1992, 2018.
[15] Zhou, Y. Wang. K. Qin, and W. Zeng. Moving Indoor: Unsupervised Video Depth Learning 2019.
in Challenging Environments. In ICCV, pages
[16] A. Ranjan, V. Jampani, I.. Balles, K. Kim, D. Sun, J. Wulff, and M. J. Black. Competitive Col- laboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation. In CVPR, pages 12240-12249, 2019.
[17] X. Chen, Ma, Wan, Li, and Xia. Multi-view 3d object detection network for autonomous driving. In CVPR, pages 1907-1915, 2017.

---

[18] C. Mitash, K. E. Bekris, and A. Boularias. A self-supervised learning system for object de- tection using physics simulation and multi-view pose estimation. In IROS, pages 545-551, 2017.
[19] A. Laddha, S. Gautam, S. Palombo, S. Pandey, and C. Vallespi-Gonzalee Myfusenet: Improv- ing end-to-end object detection and motion forecasting through multi-vien fusion of lidar data. In CVPR, pages 2865-2874, 2021.
[20] P.-H. Huang, K. Matzen, J. Kopf. N. Ahuja, and J.-B. Huang. Deepmvs: Learning multi-view stereopsis. In CVPR, pages 2821-2836 2018.
[21] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan. Mvsnet: Depth inference for unstructured multi- view stereo. In ECCV, pages 767-783, 2018.
[22] Y. Wei, S. Liu, Y. Rao, W. Zhao, J. Lu, and J. Zhou. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In ICCV, pages 5610-5619, 2021.
[23] J. Yang. W. Mao, J. M. Alvarez, and M. Liu, Cost volume pyramid based depth inference for vuewi-view stereo. In CVPR, pages 4877-4886, 2020.
[24] A. Kundu, X. Yin, A. Fathi, D. Ross. B. Brewington, T. Funkhouser, and C. Pantofaru. Virtual multi-view fusion for 3d semantic segmentation. In ECCV, pages 518-535. Springer, 2020.
[25] A. Dai and M. Nießner. 3dmv: Joint 3d ad-muuliivven view prediction for 3d semantic scene segmen- tation. In ECCV, pages 552-468, 2018.
[26] A. Ouaknine, A. Newson, P. Pérez, F. Tupin, and J. Rebut. Multi-view radar semantic segmen- tation. In CVPR, pages 15671-15680, 2021.
[27] V. Guizilini, R. Ambrus, S. Pillai, A. Raventos, and A. Gaidon, 3d packing for self-supervised monocular depth estimation. In CVPR, pages 2485-2494, 2020.
[28] C. Godard, O. Mac Aodha, M. Firman, and G. Brostow. Digging into self-supervised monoc- ular depth estimation. In ICCV, pages 3828-3838, 2019.
[29] V. Guizilini. I. Vasiljevic, R. Ambrus, G. Shakhnarovich, and A. Gaidon. Full surround mon- odepth from multiple cameras. arXiv preprint arXiv:2104.00153 2021.
[30] C. Wang, J. Miguel Buenaposada, R. Zhu, and S. Lucey. Learning depth from monocular videos using direct methods. In CVPR. pages 2022-2030, 2018.
[31] T. Shen, L. Zhou, Z. Luo, Y. Yao, S. Li, J. Zhang, Fang, and L. Quan. Self-Supervised Learning of Depth and Motion Under Photometric Inconsistency. In ICCVW, pages 0-0, 2019.
[32] L. Schonberger and J.-M. Frahm. Strecturefrrom-mottio revisited. In CVPR, pages 4104- 4113, 2016.
[33] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, pages 11621-11631, 2020.
[34] Z. Wang, L. Tang, X. Liu, Z. Yao, S. Yi, J. Shao, J. Yan, S. Wang, H. Li, and X. Wang- Orientation invariant feature embedding and spatial temporal regularization for vehicle re- identification. In ICCV, pages 379-387, 2017.
[35] J.-R. Chang and Y.-S. Chen. Pyramid stereo matching network. In CVPR, pages 5410-5418, 2018.
[36] V. Casser, S. Pirk, R. Mahjourian, and A. Angelova. Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos. In AAAI, volume 33, pages 8001-8008, 2019.
[37] B. Bozorgtabar, M. S. Rad, D. Mahapatra, and J.PP. Thiran. Syndemo: Synergistic deep feature alignment for joint learning of depth and ego-motion. In ICCV, pages 4210-4219, 2019.
10

---

[38] Y. Zou, Z. Luo, and J.-B. Huang. Df-net: Unsupervised joint learning of depth and flow using cross-task consistency. In ECCV, pages 36-53, 2018.
[39] N. Mayer, E. Ilg, Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In CVPR. pages 4040-4048, 2016.
[40] C. Zheng, T.-J. Cham, and J. Cai. T2net: Synthetic-to-realistii translation for solving single- image depth estimation tasks. In ECCV, pages 767-783, 2018.
[41] X. Fei, A. Wong, and S. Soatto. Geo-supervised visual depth prediction. RA-L, 4(2):1661- 1668, 2019.
[42] Y. Kuznietsov, J. Stuckler, and B. Leibe. Semi supervise deep learning for monocular depth map prediction. In CVPR, pages 6647-6655, 2017.
[43] C. Godard, O. Mac Aodha, and G. J. Brostow. Unsupervised monocular depth estimation with left-right consistency. In CVPR, pages 270-279 2017.
[44] F. Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, pages 1251-1258, 2017.
[45] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 170.00610 2017.
[46] D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJC,, 60(2):91-110, 2004.
[47] O. Russakovsky, J. Deng, H. Su, Krause, Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252) 2015.
[48] J. H. Lee, M.-K. Han, D. W. Ko, and I. H. Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint 190i 10326, 2019.


Reviews:
[{"review_id": "yJW5KSMTWvph", "reviewer": "Reviewer_RZGD", "summary": "This paper proposed a method that predicts high-quality depth maps across cameras by exploring correlations among surrounding views. They apply the cross-view attention to fuse the downsampled feature maps. They use structure-from-motion on adjacent frames to obtain the scale-aware pseudo depths. To exploit the pose consistency between views, they propose to predict the universal pose and then transform it to each local camera pose instead of predicting the pose of each view separately.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\nThis paper is well written and easy to follow. Their method shows good results on two datasets compared to the existing works. The experimental results show that they could effectively incorporate the information from multiple views. They provide insights to supervise the depth network to learn the real-world scale. The joint pose estimation proved to be having a positive effect on the performance. Also, the reported model’s inference time is fast.\n\nWeakness\nThe reported scores of tables 1-4 are averaged over all views. Are the performance gains consistent on each separate view?\nAs shown in table 4, the setting of w/o spatial context might lead to worse results on the metrics of Sq Rel. Also, the performance gap between w/o spatial context and the whole setting is smaller than that on the DDAD dataset. The area of overlaps between adjacent views might be different in the different datasets. Does it bring some restrictions to this method?\nFSM uses a more light-weight encoder(Resnet-18) compared to the ResNet-34 using here. It is good to see the information on the FSM’s inference time.\nThis work proposes to exploit information from multiple views to make predictions. In real applications, it is common to have extrinsic noises, one camera occluded, or camera time delay. It is good to see the robust analysis.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think results and the techinical solution are convincing and interesting. I would recommend weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please address my concerns in the weaknesses.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "VDY-gjr8XnO", "reviewer": "Reviewer_maRz", "summary": "This paper presents a novel method for multi-view depth prediction based on multiple color images without the usage of calibrated depth sensor (LIDAR). They propose extracting features with a simple backbone (ResNet34), which are combined with a Cross-View-Self-Attention method, where the final results are deconvoluted and combined with skip connections from the encoder to keep detail.\nThey show how to predict a scale for the depth estimations based on a nice pretraining on structure-from-motion data.\nFurthermore, they predict the joint pose for all cameras, showing that they even can outmatch the error in the ground truth data.\nThey also show superior results in comparison to FSM, PackNet-SfM and Monodepth2.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well written and easy to follow, something I do not say lightly\n* I especially like the idea of using an attention model for incorporating the information from the different views\n* great ablation, most questions I had were evaluated and proven to be improved\n\nWeaknesses:\n* the only thing I do not fully see proven, but it might be that I need a minor clarification, is that the method works better when it uses all images simultaneously. I feel these results are covered in table 8., but they are not fully discussed in 4.3. Can I get a clarification on this.\n* one thing I particularly do not like is when depth images are solely represented as images and not as point clouds. It is nearly impossible to judge a depth image in 2D, anyone who claims they can judge that is lying. I would have love an image or even better, a video of a point cloud, with the different metrics turned on and off to immediately see the difference\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I strongly recommend this paper for acceptance to CORL. They have proven that this paper deserves to be published in CORL. \n\n\n\nOld Review before Discussion: I will be absolutely clear in my review. I think this paper deserves a strong accept. The idea is good the evaluation is great. I just have one question, can you prove that using multiple images with the cross-view-self-attention method improves the results? If that is already hidden in table 8. please tell me, and I will gladly change my review to strong accept. If not, please add these results, and I will also change my review.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "\nNot relevant anymore: \n* more clearly show that your multi-view approach improves over using a non-multiview approach\n* add a point cloud visualization to the supplementary", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "JN6LpEgm3EX", "reviewer": "Reviewer_CgbY", "summary": "This paper aims at a unique problem of multi-camera surrounding view self-supervised depth estimation. A cross-view transformer is designed and applied at multiple scales to incorporate multi-view features. Comprehensive experiments are conducted for depth evaluation.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ This research is well-motivated for multi-view cameras on autonomous vehicles.\n+ This paper is generally well-written in a good structure.\n+ This paper mostly achieves SOTA performance in DDAD and nuScenes datasets.\n\nWeaknesses:\n- The proposed method cannot theoretically guarantee the multi-view depth consistency.\n- The computational performance is not evaluated for the robotic context.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a novel cross-view transformer to fuse multi-camera features in a multi-scale fashion for a unique problem - multi-surrounding view camera depth estimation. The application is novel, although the methodology is still considered incremental. Given the performance in this unique problem, the review suggests 'Weak Accept' or slightly below that.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- The scientific novelty and method generality should be highlighted. Since the method cannot theoretically guarantee the multi-view depth consistency, the authors are suggested to construct a few research hypotheses for the scientific novelties.\n- Joint pose estimation is included but not fully evaluated (preferred)."}}, {"review_id": "7is_Inaid0J", "reviewer": "Reviewer_nGH3", "summary": "This paper proposed a SurroundDepth module to predict the full surround depth maps from multiple cameras. \nThe major contribution includes: \n* A novel cross-view transformer with cross-view self-attention to fuse the multi-view information and enable interactions. \n* An SfM(structure-from-motion) pretraining mechanism is proposed to predict the metric scales of the depth.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* Although the full-surround depth prediction is not a new concept(e.g. FSM[29]), the author proposed a novel cross-view transformer and self-attention mechanism to better fuse the feature and enable interactions to improve the accuracy.\n* The proposed Scale-aware SfM pretraining is effective to retrieve the metric scale for depth prediction.\n**Weaknesses:**\n* The experiment results in Table3 and 4 are inconsistent with paper FSM[29], it is better to use the same evaluation settings or at least illustrate the detailed difference in evaluation.\n* It's better to also test on the KITTI dataset with the Stereo setting to better compare with the majority of the depth prediction methods.\n* The comparison of test inference time with FSM[29] is missing, what is its inference time on the same hardware setting?\n* It's better to also compare the number of parameters, consumption of GPU-RAM, and training time with the baseline methods. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The author introduced the Transformer into full-surround depth prediction task and achieved significant performance improvement according to the experiment section. However, the evaluation settings need to be further illustrated and the computational cost should be further analyzed.\n\nPost-rebuttal: \n(1)The evaluation comparison with FSM[29] should be updated. \n(2)The proposed method consumes significantly more computation and memory while the performance improvement looks marginal.\n(3)The introduction of Transformers to this task and the open-source code do contribute to the community, which is appreciated.\nBased on the above considerations, my final rating is \"weakly accept\"", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "**Major Issues:**\n* Since the FSM[29] is the only fair comparison baseline, could you use the same evaluation settings or at least illustrate the detailed difference in evaluation?\n* The Transformers are generally computational heavy, could you further compare the computational cost such as the number of parameters, consumption of GPU-RAM, and training time.\n**Minor Issues:**\n* The KITTI Monocular/Stereo performance results are appreciated if possible.\n* The limitation analysis could be more comprehensive."}}]

Author rebuttal:

Decision: Accept (poster)