Self-supervised surround-view depth estimation with volumetric feature fusion
Authors: Jung Hee Kim
Junhwa Hur
Tien Phuoc Nguyen
Seong-Gyun Jeong
Venue: NeurIPS 2022 Accept (2022)
Keywords: Surround-view depth estimation
Monocular depth
Self-supervised learning
Depth synthesis

Abstract: We present a self-supervised depth estimation approach using a unified volumetric feature fusion for surround-view images. Given a set of surround-view images, our method constructs a volumetric feature map by extracting image feature maps from surround-view images and fuse the feature maps into a shared, unified 3D voxel space. The volumetric feature map then can be used for estimating a depth map at each surround view by projecting it into an image coordinate. A volumetric feature contains 3D information at its local voxel coordinate; thus our method can also synthesize a depth map at arbitrary rotated viewpoints by projecting the volumetric feature map into the target viewpoints. Furthermore, assuming static camera extrinsics in the multi-camera system, we propose to estimate a canonical camera motion from the volumetric feature map. Our method leverages 3D spatio- temporal context to learn metric-scale depth and the canonical camera motion in a self-supervised manner. Our method outperforms the prior arts on DDAD and nuScenes datasets, especially estimating more accurate metric-scale depth and consistent depth between neighboring views.

Self-Supervised Surround- View Depth Estimation with Volumetric Feature Fusion
Jung-Hee Kim 42dot Inc. junghee kim842dot... ai
Junhwa Hur +:
Google Research
j junhwahur@google.eoo com
Tien Phuoc Nguyen'
Seong-Gyun Jeong
Hyundai Motor Group Innovation Center
42dot Inc.
tien. wgw.en@hngics.ccomm com
seonggyun, jenggyyun.joon@422oo..o
Abstract
We present a self-supervieed depth estimation approach using a unified volumetric feature fusion for surround-view images. Given of surround-view images, our method constructs a volumetric feature map by extracting image feature maps from surround-ve -vew images and fuse the feature maps into a shared, unified 3D voxel space. The volumetric feature map then can be used for estimating a depth map each surround view by projecting into an image coordinate. A volumetric feature contains 3D information at its local voxel coordinate; thus our method can also synthesize depth map at arbitrary rotated viewpoints by projecting the volumetric feature map into the target viewpoints. Furthermore, assuming static camera extrinsics the multi-camera system, we propose estimate a canonical camera motion from the volumetric feature map. Our method leverages 3D spatio- temporal context to learn metric-scale depth and the canonical camera motion self-supervised manner. Our method outperforms the prior arts DDAD and nuScenes datasets, especially estimating more accurate metric-scale depth and consistent depth between neighboring views.
Introduction
Depth perception of surrounding environment is one of the key components for 3D vision applications such as autonomous driving, robotics, or augmented reality. Specifically for autonomous driving, 3D perception benefits numerous downstream tasks including 3D objects detection 29 46 6D pose estimation 21). object tracking [8 41,, etc.
While human drivers can proactively move their heads and eyes observe their surrounding envi- ronment, an autonomous vehicle equips multiple cameras with fixed viewpoints and monitors its surroundings. The multi-camera system hence exhibits limitations subjective the fixed camera setup; the camera system might share a small portion between the adjacent viewpoints and rely on heterogeneous camera intrinsics.
As prior work, Full Surround Monodepth (FSM) [1] extended a self-supervised monocular depth estimation method [13] into multi-camers setup of autonomous vehicles. Their method exploits small image overlaps between
neighboring cameras as a supervision signal to learn metric-scale depth. Yet, FSM [15) individually estimates depth and camera motion for each view with a shared CNN, and does not utilize any additional image cues from neighboring views at test time.
denotes equal contribution. This work has been done at 42dot Inc.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) Input images (Iront-left, front Froni-right)
Focal longth manipulation (x I, 2.
(c) Yaw angle manipulation (view-interpolation hetween the left and right view)
(d) Roll angle manipulation (# 10%)
(c) Pitch angle manipulation (#3*)
Visualization of synthesized depths: (a) Given of images (e. g. front-left, front, and front-right), we can synthesize depth map at imaginary views with different (b) focal lengths (zoom-out effect), (c) yaw angles, (d) roll angles, and (e) pitch angles. Depth maps at known camera views are overlayed with input images.
Figure I:
To date, architectural design choices in the spatio-temporal domain are relatively less investigated, creating room for improvements.
We introduce a self-supervisee approach to surround- -vew depth estimation based on unified volumetric feature representation. Given multiple images covering the surrounding view, our method extracts an image feature from each image, coordinate), and fuses them into unified volumetric feature map. Each volumetric feature contains 3D information at its located voxel coordinate. Some volumetric features are shared between neighboring views due to their image overlaps; we present tailored multilayer perceptron (MLP) to process the volumetric features with consideration of the superposition. Given the volumetric features, we project the features back to the image coordinate with known camera information and pass through the depth decoder to obtain depth map at each view.
the features into 3D space (i. e., voxel
Because the volumetric feature contains 3D information its local voxel coordinate, our method can also synthesize depth map at arbitrary rotated viewpoints by projecting the features into an image coordinate with target focal length and target extrinsics. This can overcome the limitation of fixed multi multi-camera setup. Fig. Fig.Tden demonstrates the results of depth map synthesis at novel views. Our method can seamlessly synthesize depth maps arbitrary viewpoints from the volumetric features with different focal lengths and camera angles.
Furthermore, unlike previous work [15]. introduces canonical camera motion estimation, assuming static extrinsics between cameras. This geometry constraint leads to a stabilized ego- motion learning and enhanced metric-scale depth estimation. On top of that, adjust the intensity distribution for the neighboring viewpoints, which reduces irregular photometric reconstruction errors. We present self-supervised learning framework that learns estimate scale-aware depth and camera motion from unlabeled surround-vien images, using 3D spatio-temporal reconstruction.
We summarize contributions follows: (i) we introduce novel volumetric feature representation that effectively learns to estimate the surround-view depth and canonical camera motion. To our knowledge, method is the first approach that targets multi-camera feature fusion with self- supervised depth estimation. (ii) We demonstrate synthesizing depth maps at arbitrary views with the unified volumetric feature representation. (iii) On popular multi-camera datasets 2 14], our method
demonstrates consistent improvements compared to the state-of-the-art method 15
Related Work
Monocular depth estimation. Supervised-learming leandd-eeaannnneeeeee based monocular depth approaches g 10 20
22 36
require large amount of annotated data for supervision, and as a result, limits the scalability of their methods due to the expense of acquiring such annotated data on diverse scenes. To address the limitation, self-supervised methods propose proxy learning tasks that use unlabeled temporally- consecutive images 3 14|
23
24 37
49
51 52
stereoscopic pairs 13 25 34 35
during

---

Input images
Surround view feature fusion
Depth and moñon decoding
Outguts
Canorical mobion esiimation
Camn 2 TT
Image crcodor
Velameete::
featiee enco
decoder
N: T
Wilnntric
feature
Distribute local
Source images 1
Depth estimation
Image erendder
Vilumetiic
Volumebric feature
Proected fratures
Target Images 1
Target imagge depill.
encoder
Volumetric feature
Secrn map
Image
Over region MLP
MLF
F
Non up oegion MEF
Velument:
Uipeample
feature map
Figure 2: Architecture overview: Our method first extracts image feature maps from surround-view images and fuses them into into a volumetric feature map by considering overlapped fcatures hetween neighboring views. The depth estimation module projects the volumetric feature into cach view and estimates per-view depth map. The canonical motion estimation module flattens the volumetric feature along the z-axis and outputs the canonical motion. The output canonical motion can distributed each view as local motions.
training time. For supervision signals, the approaches minimize image reconstruction errors between reference images and synthesized images that generated from output depths and temporally-(or images. Yet, the methods output depth maps only up to scale or at a fixed viewpoint. In this work, demonstrate surround-view depth method that outputs depth maps arbitrary viewpoints as well as in metric seale.
Omnidirectional depth estimation. Previous work has demonstrated estimating depth on wide field-of-view (FOV) images (48] 360 images 45 44 53) that covers surrounding view, and efforts on finding a suitable projective geometry to process those input images have been continued. directly used 360 images that are represented under equirectangular projection, but there exists severe visual distortion on the top and bottom parts of the images. To avoid such distortion on the 360 image, cubemap representation 4 44 has been presented. which rectifies input 360" image into cube map coordinate and uses it for the input instead. Won al. (48) proposed to directly perform a stereo matching of multiple fisheye images in a spherical coordinate. Neural Ray Surfaces [42] introduced generic framework that jointly learns to estimate depth and motion without prior knowledge of specific camera models.
Initial approaches 44 53
Volumetric feature representation. The basic idea of fusing multi-view features into a shared voxel space has been presented in other tasks, such as multi-view stereo [19] 26
30
32
50 or
3D semantic segmentation [5] 26]. The approaches mainly focus on solving a matching task for object-level or static indoor scene reconstruction using intractable 3D convolution. Based on the similar principle, our method presents a volumetric feature representation for surround-view depth and canonical motion estimation. Comparing to 3D convolution, we demonstrate that light- weighte MLP layers are more efficient for fusing features from surround-vier images with small overlaps.
Some approaches [17 28 33 38 demonstrate aggregating 2D features from surround-view images onto a Bird's-eye-view (BEV) space for semantic segmentation or object detection. However, the BEV-based representation does not precisely preserve 3D information due to the abstraction of the height information. contrast, our approach presents a voxel-based representation that is more suitable for depth and canonical pose estimation.

---

3 Surround-View Depth Estimation via Volumetric Feature Fusion
Given two temporally consecutive surround-view images, I and I captured by multiple cameras C, (i E {1, 2,..., 6} in our setup) that cover a surrounding view of a vehicle, our method estimates metric-scale depth D for each image I from each camera C₁ (but also at an arbitrary rotated viewpoint) and a canonical camera motion of the vehicle T T as an auxiliary task. Our methods assumes known camera intrinsics K, and extrinsics E,, but each camera can have different intrinsics.
3.1
Architecture overview
We propose novel fusion module for murround-vient depth estimation. Fig. shows overview of our approach, consisting of the surround- view volumetric feature fusion (Sec. 3.2). depth estimation (Sec. 3.3 and canonical motion estimation (Sec. 3.4 module.
The surround-vies view feature fusion module constructs a single, unified volumetric feature map V by aggregating multi-scale image feature maps F that are extracted from each image in the surround- camera setup. From the volumetric feature map V. the depth estimation module retrieves per-frame feature that represents specific viewpoint and passes it through depth decoder to produce a depth map for the corresponding view. Because a volumetric feature each voxel contains its local 3D information, method also synthesize depth map at arbitrary view by projecting the volumetric fcature map into the target view. The canonical motion estimation module flattens the volumetric feature map along the z-axis and predicts global motion referring to the canonical camera coordinate. In contrast to FSM [15] that independently estimates each camera motion, our design globally reasons camera motion, which not only reduces the number of unknown factors but also makes the problem simpler.
3.2
Surround-vic volumetric feature fusion
Image feature encoding. We first extract image features from surround-vied images. Given a set of surround-vies images, pass each image I, through shared 2D image encoder obtain image feature map F, for 7th view. We use ResNet [16] as the encoder which takes an image I, (with resolution of H W) and outputs a multi-scale feature pyramid. We take the last three feature maps from the feature pyramid, reshape them up to resolution of !! W apply 1x1 convolution for channel reduction. As result, obtain single feature map F; for each image, where the feature map conveys multi-scale and dighndimensional information of the image.
3 x ". concatenate them, and
Volumetric feature encoding. We aggregate the image feature maps F, from surround- view images into a single, unified volumetric feature map V in pre-defined voxel space, illustrated in Fig. B For each voxel, we find corresponding pixel p(w, h) by projecting the voxel (r, y: 2) into pixel coordinate. Then, bilinearly interpolate image feature (()) at the projected pixel p(w, h) to handle sub-pixel location and allocate the sampled image feature to the voxel. Because the sampled image feature F(p) contains high-level information along its pixel ray, we extract local 3D feature at the voxel (x, z) by concatenating the feature with a depth value of the voxel (i. e., depth positional encoding) and pass through an MLP. In this way, voxels that get through by single pixel ray have their individual 3D feature extracted by the MLP instead of having the same image feature.
F((p.) di
Volumettic feature
Doerlap fextures
Non overtap features
P((p)) do
PC
MUP for overlap
Py(ps) de
Volumetric:
feature
Depth
Bilinear feature interpolation each vosel
positional encoding
MLP for nun-overlpp
Surround volumetric feature
Figure 3: Surround-view volumetric feature fusion: ing image features from each view. After the depth positional encoding, we pass the features through MLPs to encode volumetric feature. We use two different MLPs for overlap and non-overlap region.
For each voxel, we first bilinearly interpolate correspond-

---

Some voxels are associated with multiple image features due to spatial overlaps across the camera views. To merge the multiple features on overlap region, we use the other MLP that learns fuse the multiple per-pixel image features into per-voxel volumetric feature.
3.3 Depth estimation
From the unified volumetric feature map, aim to estimate depth map of each surround-view input image but also synthesize depth maps at any arbitrary rotated camera viewpoints.
Transformation of volumetric feature as projected image feature. For each target view C, that we want to estimate its depth map, we project the volumetric feature map V into the projected image feature map F, with resolution of и W using camera extrinsics E; of the target view. For each pixel p at the target image coordinate, we uniformly sample volumetric features along its pixel ray and concatenate the sampled features obtain the projected image feature (p)).
Depth decoder. To predict a depth map from the projected image feature map F,, we use a light- weighted depth decoder Decoopt consisting of convolutional layers: 3 convolutional layers for H W) and one convolutional layer for the depth output.
upsampling #
Depth map synthesis at novel view. We further extend our depth estimation module to produce depth map for a novel viewpoint which is not covered by the original multi-camera system. Because volumetric feature each voxel encodes local 3D structure information in certain range, our method can synthesize a depth map at a desired arbitrary viewpoint by projecting the volumetric feature into the viewpoint and passing through the depth decoder. Fig. [ visualizes synthesized depth maps at arbitrary rotated viewpoints. Our method can seamlessly synthesize depth maps with variations of yaw, roll, or pitch angles as well as arbitrary focal lengths.
3.4 Canonical motion estimation
The prior work, FSM [5), separately estimates a relative camera motion of each camera using a shared encoder-decoder network. However, this process can be redundant because multi-camera system that is attached to a vehicle shares the same canonical motion. To improve, we introduce a canonical pose estimation module that takes collapsed volumetric feature map FEEY and predicts one representative canonical camera motion as illustrated Fig. 2 Given the estimated canonical camera motion, Here, we use the same architecture design to obtain a volumetric feature map (cf. Sec. Sec.3.2) but with different trainable network weights, and use two temporally consecutive images for the input.
direct compute the motion of each camera with their known camera extrinsics.
Volumetric feature reduction. Given a surround-vige volumetric feature map V E X we flatten the feature map onto Bird's S-Eye- View (BEV) shape (see Fig. 2 We collapse the Z dimension into the channel dimension C (i. e., reshaping it into 3D tensor
Rxxxx(2-0)) and
apply 2D convolutions to reduce channel of the feature, resulting in a 3D tensor of F BEV E xXXYXC' This collapsed volumetric feature map F BEy contains information on canonical ego-motion that is aggregated from surrounding views. Then we use the standard pose decoder from the PoseNet to estimate the canonical camera motion T6-(+1
Computing local camera poses. Assuming a static relationship between the cameras, we distribute the predicted canonical camera motion each camera motion. From the given extrinsic E, of each camera C₁ and canonical camera motion T1-1+11 we compute each camera motion as:
E E
E E,
where E, is the extrinsic of the canonical camera motion. Note that any viewpoints could also be a canonical motion. In this work, we define the canonical motion as the motion of the front-view camera.
5

---

3.5 Self-supervised learning
Given the camera motion and depth map per each view, we apply our self-supervised loss given temporal triplet of surround-view images. At test time, our method only requires images at the current time step t.
Self-supervised loss. Our self-supervised loss consists of the image reconstruction loss Liny and the depth synthesis loss Leepth
-= Limg Laepth-
The image reconstruction loss penalizes the reconstruction error between the reference image I' and the synthesized images Ie+ from each temporal, spatial, and apatio-tempora context (cf. Eq. (4)
Ling Li + Ap + 1ppsp + Asanoothhsmooll
Each term stands for the temporal L. spatio Cup spatio-temporal L.p and smoothness loss Lmmooth- To measure the reconstruction error, similarity
weighted of intensity difference and structure 12 13 47 (1 --))]| - +a 1 1-SSS-(1'' 1++1 with a = 0.85. The smoothness term 인 Ep Emez, V&D -I]ll
Csmooth follows an edge-aware Ist-order smoothness: Lsmooth
One main difference to the baseline work [5] that we do not use the pose consistency loss because method outputs representative camera motion for all views. This reduces the number of unknowns and thus encourages convergence and better accuracy.
Spatio-temporal context. The core idea of the self-supervised proxy task [15 exploit small overlaps between spatially or/and temporally neighboring images for matching to provide supervi- sionary signal and scale information for camera motion and depth. For reference image I from camera C at time step f, it synthesizes reconstructed images Ī from (i) spatial neighboring images Iý (j: indices of neighboring cameras), (ii) temporally neighboring images I (t' E (t 1, 1.tt11111 1}), and (iii) spatio-temporally neighboring images I given the predicted camera motion. A pixel-wise warping operation for the image reconstruction is defined as:
I1 K2 D.K,"
(4a)
with
T (t - 1, E,E E, I' " ( 11-1,,10 1,1+,1+1]
for temporal context, for spatio context, for spatio-temporal context,
X ij
(4b)
with estimated depth D; and camera intrinsics K, and K,. Here, Xij is the camera motion between the two cameras depending on their spatio-temporal context.
By minimizing the photometric difference between the reference image and reconstructed image, self-supervised loss function guides the network to output metric-scale depths and camera motions that can satisfy the overall contexts.
Depth synthesis loss. novel view:
We introduce a depth synthesis loss for the successful depth synthesis at
Edepth Axgons Ccons Addethhssmood ClapehhssoocthcCcch..o.oooo Adopth
(5a)
Leons N DDD:
(5b)
fedepth smooth = + (V.,
(5c)
The depth consistency loss Leves penalizes the depth difference between the synthesized depth D at the novel view and depth D, each known camera view i, adopted from Bian et al. [1]. The smoothness loss Edepth_smonne smooth 27 encourages the 1st-order smoothness of the synthesized depth map
Dj at the novel view so that it regularizes set of volumetric features retrieved at the novel view to improve 3D-awareness at each voxel coordinate. For the synthesis, we augment depth map at novel view for each known camera view i and apply the depth synthesis loss.

---

iraga war ping
Al Aighmmntt
reconstructin loss
Target image (ront IST viow)
Neconstructed
Intendhy-eligeed
Reforence image vicot
image I
iage Liga
Image intensity alignment
alignment
(c) the aligament
Figure Intensity distribution alignment: (a) After warping target image, we align mean and variance
of the reconstructed image to match with reference image's. s. (b) When aligning the distribution, it resolves the artifacts on the depth, which appears (c) when training without the alignment.
Experiments
4.1 Implementation details
Dataset and evaluation protocol. We use the DDAD 14 and nuScenes dataset for our experiments. Both datasets provide surroundview images from total of 6 cameras mounted a vehicle and LiDAR point clouds for the depth evaluation. We train our model on each train split and report the accuracy the test split. During training, the input images resolution of 384 640 for the DDAD dataset, and that of 352 x 640 for the nuScenes dataset. We train model the DDAD dataset for 20 epochs and the nuScenes dataset for 5 epochs.
down-sampled to
For evaluation, we follow the same protocol from FSM [15 that evaluates depth up to 200 (m) for the DDAD dataset and 80 (m) for the nuScenes dataset. We use conventional depth evaluation metrics proposed by Eigen et al. g We provide details of the metrics the supplementary material.
Training. We implemented our networks in Py Torch [31] and trained on four A100 GPUs. For the
image encoders, we used ResNet- 18 with weights pre-trained on ImageNet [39] All experiments used the same training hyper-parameters (unless explicitly mentioned): Adam optimizer with |3 = 0.9 and 10-1 decaying at ] of the entire training schedule with factor of 0.1; the previous t 1 and subsequent + 1 images are used as temporal context at the training time. Note that our depth estimation module does not require temporal contexts, and therefore temporal contexts are not included in the test time. For our volumetric feature, we used voxel resolution of (1m, Im, 0.75m) with spatial dimensions of (100, 100, 20) for (x, y, z) axis respectively. We color jittering as data augmentation. For the depth synthesis loss, we use the random rotation with a range between [-59, -5", -25% and [5", 50 25%] for the depth map synthesis at novel view. In the self-supervised loss in Eq.
B3 0.999; mini-batch size of per each GPU and learning rate 440 with
depth smoothness weight Asmouth = 1 x 10-3, spatio loss weight Ay = 0.03, spatio-temporal weight Xup 0.1, depth consistency weight Acoms 0.05, and depth smoothness weight novel views Adopth Adepth_smooth smooth 0.03.
The DDAD dataset [4] includes images with different focal lengths; in order to train the network to output consistent depth scale regardless of different focal lengths, we use the focal length normalization [9 which normalizes the scale of output depth to default focal length.
Intensity distribution alignment. We observe that different lighting conditions in neighboring images yields artifacts on the depth output (Fig. Fc) with the self-supervised loss in the previous work [5]. To address the issue, we align mean and variance on overlapped regions of the reference image I and synthesized image I:
Laign (i- 0/a 14.
where {f, are the mean and variance of the reference image I. and (jz, that of the synthesized image i. The aligned image Talign is then used for the loss calculation in Eq. 2 Fig. 4a briefly illustrates this process. Here, the reconstructed image can be from either a spatial, temporal, or spatio-temporal context.
4.2 Surround-view depth evaluation
We compare the accuracy our method with state-of-the-art methods. Especially for our closest baseline FSM [15), we use reproduced version of FSM for the comparison because the
7

---

Figure 5: Our results on DDAD dataset: Our method successfully estimates depth at each view via volumetric feature fusion.
Table 1: Evaluation on the DDAD dataset [14) and nuScenes dataset [2] We compare our method with the state of the arts. We report the averaged accuracy from all views using the metric from Eigen et al. [ Median-scale accuracy ground truth for correctly scaling the depth map during evaluation (8.
Dataset Scale Method
Abs Rel
Sq Rel
RMSE
RMSE log
5 1.25
1.25
1.25
Monodepth2
L
0.362
14.404
14.178
0.412
0.683
0.859
0.922
Medion
PacKNer-SiM
4
0.301
5.339
14.115
0.395
0.624
0.828
0.908
DDAD
Our reproduced FSM
1
0.219
4,161
13.163
0.327
0.703
0,880
0.940
Ours
0.221
3.549
13.031
0.323
0.681
0.874
0.940
Meri
Our reproduced FSM
0.228
4.409
13.433
0.342
0.687
0.870
0,932
Ours
0.218
3.660
13.327
0.339
0.674
0.862
0.932
Monodepth2
[
0.287
3.349
7.184
0.345
0.641
0.845
0.925
nuSeenes
Median
Pa
14
0.309
2,891
7.994
0.390
0.547
0.796
0.899
Our reproduced FSM
0.301
6.180
7.892
0.366
0.729
0.876
0.933
Ours
0.271
4,496
7.391
0,346
0.726
0.879
0.934
Merrc
reproduced
0.874
0.931
Ours
0.289
5.718
7.551
0.348
0.709
0.876
0.932
implementation is not provided. For the reproduction of FSM [13]. found that the intensity distribution alignment and focal length normalization, which we explained above, were critical for efforts, there is still gap between reproduced version
stable self- superveed training. Despite and the reported accuracy; for a fair comparison, we conduct experiments under the same training and evaluation settings and demonstrate accuracy gain over the reproduced baseline.
the evaluation on both DDAD 14 and nuScenes datasets. We also report the accuracy using per-frame median scaling for the reference. On the DDAD dataset, our method shows competitive or better accuracy to the bascline method FSM [15] on 5 out of metrics. Especially, method substantially improves the accuracy the Sq Rel metric by around 17%. On nuScenes dataset, our method demonstrates consistent improvement over the baseline, by outperforming the baseline 6 out of 7 metrics, with over 24% improvement on the Sq Rel metric. Better accuracy also observed when using the median scaling for evaluation. We further demonstrate point cloud reconstruction visualization in Sec D.3 of supplementary material to provide clear evidence of the strength of our method.
[
4.3 Ablation study
Surround-vied volumetric feature fusion. To validate our main contributions, we conduct an ablation study of our volumetric feature fusion and canonical motion estimation module over the FSM baseline [15]. We train the model on DDAD dataset [14] while keeping the original experiment setup. As shown in Table 2 both of our contributions consistently improve the accuracy over the baseline. Our volumetric feature representation embeds effective 3D information and results in better
depth accuracy; the main difference from previous works [13 [5] is few additional multilayer

---

(d) Canonical
(a) Input image
(b) Overlapped region
(c) Baseline
Our fall model
estimation
Figure 6: Qualitative comparison from our ablation study: (a) Given set of images (e. g., rear-left, rear, and rear-right), (b) estimated overlapped regions from nearby cameras, (c) depth results from our reproduced baseline, (dl depth results using the baseline model with canonical motion estimation module, and our final results using both volumetric feature fusion and canonical motion estimation module.
Table 2: Ablation study on our volumetric feature fusion and canonical fusion modules: Our volumetric feature fusion scheme and canonical motion estimation module consistently improve the depth accuracy, substantially Abs Rel and Rel metries.
Volumetric feature lusion
Canonical motion
Abs Rel
Sq Rel
RMSE
RMSE log
1.25
< 1.25"
5 1.25
(Our repraduced bavelino)
0.228
4.409
13.433
0.342
0.687
0.870
0.932
0.222
3.969
13,492
0.344
0.677
0.865
0.931
0.218
3,660
13.327
0.339
0.674
0.862
0.932
Table 3: Comparison result of different feature encoding methods: To aggregate surround-view image features into voxel space, compare three different ways, averaging features (Avg voxel), applying 3D convolution layers (3D conv), using MLPs (Proposed). Our proposed approach using MLPs demonstrates better depth accuracy than the other methods.
Depth fusion
Abs Rel
Sq Rel
RMSE
RMSE log
6 1:25
6 1.252
8 1.25
Avg voxel
0.224
4,150
13.464
0.346
0.672
0.865
0.931
3D conv Proposed (MLPS)
0.228
4.179
13.474
0.348
0.680
0.862
0.924
0.218
3.660
13.327
0.339
0.674
0.862
0.932
perceptrons with volumetric feature representation while keeping the similar conventional encoder and decoder networks. Furthermore, we show the benefit of the volumetric feature representation for the camera motion module; globally reasoning the camera motion benefits the depth estimation. Fig. [6] further provides qualitative results of each setting in the ablation study and demonstrates qualitative gains of each contribution.
Volumetric feature encoder. Table 3 shows the comparison result using different approaches to aggregate surround-view image features into shared voxel space. We compare three different methods that are commonly used to process features in the voxel space. Avg voxel simply averages features that are accumulated at the same voxel coordinate. 3D conv applies sequential 3D convolutional blocks to the volumetric feature. Each 3D convolutional block composed of a 3D convolutional layer, batch normalization, and LeakyReLU, in consecutive order. Our method, using MLPs, clearly outperforms the rest two methods. Compared to 3D convolution, we find that MLPs more effectively learn pool weight multiple features overlap regions.
In Fig. we further visualize error maps each method (Avg voxel, 3D conv, MLPs) including the baseline, FSM [5] The results provide clear evidence that the volumetric feature approach takes full advantage of overlap regions compared to the [55. Furthermore, our MI.P-based method shows the highest accuracy in overall regions including moving objects, compared to Avg voxel or 3D conv.

---

Images
Overlap
FSM
Avg voxe!
3D cONV
Proposed
Figure 7: Comparison on different volumetric feature encoding methods: Qualitative comparison result using Abs Rel error map to visualize the model performance by regions. Methods using volumetric features (Avg voxel, 3D conv, Proposed) show accuracy improvement especially overlap regions compared to FSM [5]
Conclusions
We proposed novel volumetric feature representation for supe-supe surround-vier -iew depth estimation. The volumetric feature representation aggregates image features from surround-veww images, encodes 3D information, and then used to estimate depth map each view and canonical motion between two temporally consecutive images. Furthermore, by projecting the volumetric feature arbitrary rotated views, method can also synthesize depth map the novel view, additionally with varying focal lengths. Our method outperforms the state-of-the-art surround-view depth method. The ablation study successfully validates our contributions well design choices on the volumetric feature encoding. In future work, we expect our volumetric feature representation can benefit other computer vision tasks such object detection, image segmentation, and motion estimation in the surround-view camera setup.
Limitation. Our volumetric feature representation is defined in a 3D cubic space; it needs to take the trade-off between memory usage and the resolution into consideration. Because we focus on embedding 3D information to novel representation, sometimes some blurred regions occurs in the depth map (e. g., in Fig. and Fig. 6). The usage of cylindrical or spherical coordinates would improve the efficiency of the memory usage. Furthermore, some artifacts sometimes appear in the synthesized depth map in Fig. [ where no input image feature is given. In future work, we will consider using additional regularization terms to preserve details of depth results as well reduce the artifact.
Potential negative societal impacts. Methods for surround-view depth estimation require large number of images for training, which may include images with privacy issues. To prevent this, the dataset should be precisely curated and be used for the research purpose only.
10

---

References
[1] Jiawang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, and Ian Reid. Unsupervised scale-consistent depth and ego-motion learning from monocular video. In NeurIPS. 2019.
[2] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. In CVPR, pages 11621-111 11631, 2020.
[3] Vincent Casser, Socren Pirk, Reza Mahjourian, and Anelia Angelova. Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos. In AAAI, pages 8001-8008, 2019.
[4] Hsien-Tze Tzu Cheng, Chun-Hung Chao, Jin Jin-Dong Dong, Hao-Kai Wen, Tyng-Luh Liu, and Min Sun. Cube padding for weakly- supervie saliency prediction in 360 videos. In CVPR, pages 1420-14400 14200-112,,200 2018.
[5] Angela Dai and Matthias Nicßner. 3DMV: Joint 3D-multi-view prediction for 3D semantic scene segmen- tation. In ECCV, pages 452-468, 2018.
[6] Yan Di, Fabian Manhardt, Gu Wang, Xiangyang Ji, Nassir Navab, and Federico Tombari. SO-Pose: Exploiting self-occlusion for direct 6D pose estimation. In ICCV, pages 12396-12240, 2021.
[7] Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang, Jianping Shi, Zhiwu Lu, and Ping Luo. Learning depth-guided convolutions for monocular 3D object detection. In CVPR, pages 16772-11681, 2020.
[8] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In NIPS, 2014.
[9] Jose M. Facil, Benjamin Ummenhafer, Huizhong Zhou, Luis Montesano, Thomas Brox, Javier Civera. CAM-Convs: Camera-aware multi-scale convolutions for single-view depth. In CVPR, pages 11826-1 1835, 2019.
[10] Huan Fu, Mingming Gong, Chaohui Wang. Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In CVPR, pages 2002-2011, 2018.
[11] Ravi Garg, Vijay Kumar Bg, Gustavo Carneiro, and Ian Reid. Unsupervised CNN for single view depth estimation: Geometry to the rescue. ECCV, pages 740-756. 2016.
[12] Clément Godard, Oisin Mac Aodha, and Gabriel Brostow. Unsupervised monocular depth estimation with left-right consistency. In CVPR, 2017.
[13] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J. Brostow. Digging into self-supervised monocular depth estimation. In ICCV, pages 3828-3838, 2019.
[14] Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon. 3D packing for self-supervised monocular depth estimation. In CVPR, pages 2485-2494, 2020.
[15] Vitor Guizilini, Igor Vasiljevic, Rares Ambrus, Greg Shakhnarovich, and Adrien Gaidon. Full surround monodepth from multiple cameras. RA-L, 2021.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[17] Anthony Hu, Zak Murez, Nikhil Mohan, Soffa Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. FIERY: Future instance prediction in bird's-eye view from surround monocular cameras. In ICCV, pages 55273-15282 2021.
[18] Jan Issac, Manuel Wüthrich, Cristina Garcia Cifuentes, Jeannette Bohg, Sebastian Trimpe, and Stefan Schaal. Depth- based object tracking using robust gaussian filter. In ICRA, 2016.
[19] Abhishek Kar, Christian Häne, and Jitendra Malik. Learning a multi nie stereo machine. In NIPS, 2017.
[20] Laina, Christian Rupprecht. Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 3DV, pages 239-248, 2016.
[21] Tuan- Tuan-Tang Le, Trung-Sor Son Le, Yu- Ru Chen, Joel Vidal, and Chyi- Yeu Lin. 6D pose estimation with combined deep learning and 3D vision techniques for a fast and accurate object grasping. Robotics and Autonomous Systems, 141:103775, 2021. 2021.
11

---

[22] Jin Han Lee, Myung-Ko Kyu Han, Dong Wook Ko, and II Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. In urXiv:1907.10322 [cx. [cs.CV], 2019.
[23] Chenxu Luo, Zhenheng Yang, Peng Wang. Yang Wang, Wei Xu, Ram Nevatia, and Alan Yuille. Every pixel counts++: Joint learning of geometry and motion with 3D holistic understanding. TPAMI, 2019.
[24] Reza Mahjourian, Martin Wicke, and Anelia Angelova. Unsupervised learning of depth and ego-motion from monocular video using 3D geometric constraints. In CVPR, pages 5667- 0675, 2018
[25] Ishit Mchta, Parikshit Sakurikar, and Narayanan. Structured adversarial training for unsupervised monocular depth estimation. In 3DV, pages 314-323, 2018.
[26] Zak Murez, Tarrence van As, James Bartolozzi, Ayan Sinha. Vijay Badrinarayanan, and Andrew Rabinovich. Atlas: End-to-end 3D scene reconstruction from posed images. In ECCV, pages 414
[27] Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan. RegNeRF: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, 2022.
[28] Bowen Pan, Jiankai Sun, Ho Yin Tiga Leung, Alex Andonian, and Bolci Zhou. Cross-view semantic segmentation for sensing surroundings. RA-L, 5(3):4867 4879)4837777 2020.
[29] Dennis Park, Rares Ambrus. Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular 3D object detection? ICCV, 2021.
[30] Despoina Paschalidou, Osman Ulusoy, Carolin Schmitt, Luc van Gool, and Andreas Geiger. RayNet: Learning volumetrie 3D reconstruction with ray potentials. In CVPR, pages 3897-3906, 2018.
[31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In NewrlPS, pages 8024-8035, 2019.
[32] Rui Peng, Rongjie Wang, Zhenyu Wang, Yawen Lai, and Ronggang Wang. Rethinking depth estimation for multi-view stereo: A unified representation. In CVPR, pages 8645 8654, 2022.
[33] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D. In ECCV, pages 194-210, 2020.
[34] Sudeep Pillai, Rareş Ambrus, and Adrien Gaidon. SuperDepth: Self-supervised. super-resolved monocular depth estimation. In ICRA, pages 9250-9256, 2019.
[35] Matteo Poggi, Fabio Tosi, and Stefano Mattoccia. Learning monocular depth estimation with unsupervised trinocular assumptions. In 3DV, pages 324-333, 2018.
[36] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, pages 12179-12 12188, 2021.
[37] Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun. Jonas Wulff, and Michael J. Black. Competitive collaboration: Joint unsupervised learning of depth, camera motion. optical flow and motion segmentation. In CVPR, 2019.
[38] Cody Reading, Ali Harakeh, Julia Chae, and Steven I.. Waslander. Categorical depth distribution network for monocular 3D object detection. In CVPR, pages 8555-8564, 2021.
[39] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. IJCV, 115(3):211 252, 2015.
[40] Leslie N. Smith and Nicholay Topin. Super-Convergence. Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications. volume 1 11006, page 1100612, 2019.
[41] Ales Leonardis Song Yan, Jinyu Yang and Joni-Kristian Kamarainen. Depth-only object tracking. In BMVC, 2021.
[42] Igor Vasiljevic, Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Wolfram Burgard, Greg Shakhnarovich, and Adrien Gaidon. Neural ray surfaces for self supervised learning of depth and ego- motion. In 3DV, pages 1-11, 2020.
12

---

[43] Fu-En Wang, Hou-Ning Hu, Hsien-Tzu Cheng, Mian-Ting Lin, Shang- Ta Yang. Meng-Li Li Shih, Hung kuo Chu, and Min Sun. Self-supervised learning of depth and camera motion from 360 videos. In ACCV, 2018.
[44] Fu- En-En Wang, Yu-Hsuan Yeh, Min Sun, Wei-Chen Chiu, and Yi-Hsuan Tsai. BiFuse: Monocular 360 depth estimation via bi-projection fusion. In CVPR, pages 459-468. 2020.
[45] Ning-Hsu Wang, Bolivar Solarte, Yi-Hsuan Tsai, Wei-Chen Chiu, and Min Sun. 360SD-Net: 360 stereo depth estimation with learnable cost volume. In ICRA, pages 582 588, 2020.
[46] Yan Wang. Wei-Lun Chao, Divyansh Garg. Bharath Hariharan, Mark Campbell, and Kilian Q. Weinberger. Pseudo-LiDAR from visual depth estimation: Bridging the gap object detection autonomous driving. In CVPR, pages 8445-8453, 2019.
[47] Jamie Watson, Oisin Mac Aodha, Victor Prisacariu, Gabriel Brostow, and Michael Firman. The Temporal Opportunist: Self-supervised multi-frame monocular depth. In CVPR, pages 1164-1141 1174, 2021.
[48] Changhee Won, Jongbin Ryu, and Jongwoo Lim. OmniMVS: End-to-end learning for omnidirectional stereo matching. In CVPR, pages 8987-8996, 2019.
[49] Zhenheng Yang, Peng Wang. Yang Wang. Wei Xu. and Ram Nevatia. LEGO: Learning edge with geometry all at once by watching videos. In CVPR, pages 225-234, 2018.
[50] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. MVSNet: Depth inference for unstructured multi-view stereo. In ECCV, pages 767-783, 2018.
[51] Zhichao Yin and Jianping Shi. GeoNet: Unsupervised learning of dense depth, optical flow and camera pose. In CVPR, 2018.
[52] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, pages 1851-1858, 2017.
[53] Nikolaos Zioulis, Antonis Karakottas, Dimitrios Zarpalas, and Petros Daras. OmniDepth: Dense depth estimation for indoors spherical panoramas. In ECCV, 2018.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Sec. T
(b) Did you descrihe the limitations of your work? [Yes) See Sec. 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See Sec. and additional information is described in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 4.1
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] The experiments were done using the fixed seed value to reduce reproducibility issue. Further, code is provided in the supplementary material
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Sec. 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Sec. we use public research dataset, we' ve cited their works.
(b) Did you mention the license of the assets? [Yes] In the supplementary materials, we include the license condition code basis and public datasets that we used.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We include our implementation in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [No] We've ve used public dataset and cited there work in the paper. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] Public surround-vien depth datasets do not contain any personally identifiable information or offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] Crowdsoureing human subjected research was not included in this work.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "eeaQyO9nwVz", "reviewer": "Reviewer_9Ef1", "summary": "This paper proposes a novel surround-view depth estimation, and project features of different views into a volumetric feature spaces, where overlapped views can enhance each other and get better depth maps. The pipeline also enables novel-view depth synthesis. The problem setting may be useful for autonomous driving. \n\nThe experiments are insufficient. ", "questions": "- Is it possible for the proposed method to compare with monocular methods? There are many common benchmarks for monocular depths. \n- Would the cubic space introduce some artifacts and distortions comparing to spherical spaces? \n- As a self-supervised method, I assume there is no GT depth needed? Is that so? Is it possible to train on real data without GT? ", "limitations": "Yes. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Paper is easy to follow. \n+ The problem setting is novel. \n\n- The depth map quality is on par with monocular methods. \n- No visualizations of different methods on DDAD and nuScenes are provided. From Figure 5, the depth maps are similar to existing monocular methods in quality. \n- In Table 2, depth fusion is not evaluated individually. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "J8IkkUK18Vd", "reviewer": "Reviewer_ZBww", "summary": "This paper introduces a self-supervised depth estimation method based on a unified volumetric feature representation encoded from surround-view. The proposed method consists of three parts. First, the surround-view feature fusion module generates a unified volumetric feature from the extracted multi-view 2D image features. Second, the depth fusion module reconstructs the depth map given an input camera viewpoint. Last, the global motion of the canonical camera is estimated by assuming static camera extrinsic. Experiments on DDAD and nuSecens datasets achieved improved results over existing methods.\n", "questions": "- More details for the construction of the volumetric feature.\n- Effect of volume resolution on memory consumption and depth estimation.", "limitations": "The authors have discussed the limitation and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea of using a volumetric feature to encode information from surround-view images seems interesting and logical.\n- With the construction of volumetric feature and depth fusion, the proposed method achieves improved results on DDAD and nuScenes datasets.\n- Ablation study has been provided to verify the effectiveness of depth fusion and canonical motion estimation.\n\n\nWeaknesses:\n- The core of this method is the construction of the unified volumetric feature to estimate the surround-view depth and canonical camera motion. However, some details for constructing a unified volumetric feature are missing. This method simply copies the image feature to all voxels went through by the corresponding pixel ray. In this case, some voxels may have been passed by multiple pixel rays. It is unclear how to handle multi-feature collisions.\n- Related to the first question, why different MLPs are used to fuse the overlap features and non-overlap features in the volumetric feature (see Line 132)? No experiments are shown to verify this strategy.\n- Since this method encodes the feature in a 3D volume, the effect of volume resolution on the memory consumption and depth estimation should be discussed.\n\nOverall I think the proposed idea is interesting, but I have some questions for the volumetric feature construction part. I would give a borderline accept rating at this stage.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9zUdxDj40y", "reviewer": "Reviewer_K314", "summary": "The authors propose a self-supervised approach for the task of surround-view depth estimation. 2D image features are un-projected onto a shared 3D volume, which is later queried for decoding depth maps at target views. Similarly, pose changes of the canonical frame is also decoded from volumetric feature pooled from all 2D views. Evaluations conducted on DDAD and nuScenes show competitve results compared to FSM and others. ", "questions": "It's important to address the closely related prior work mentioned above and highlight any new contributions. \n\nAlso, I feel the paper needs to have a more convincing analysis of the differences compared to FSM and others, e.g. examples highlighting typical qualitatively improvements. ", "limitations": "Adequately addressed. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__\n\nBecause of the use of a volumetric feature representation, features from images are aggregated in a shared space. This should help produce more consistent depth maps across views. \n\nAlso because of the use of a volumetric feature representation, the proposed model has the unique capability of predicting depth maps for views not among the inputs. \n\n\n__Weaknesses__\n\nA few relevant works are not included or discussed sufficiently. The way 2D features are un-projected onto a shared 3D volume aggregation has been used in many places, e.g. [1], [2], [3]. \n\n[1] Atlas: End-to-End 3D Scene Reconstruction from Posed Images, ECCV 2020. \n[2] Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D, ECCV 2020.\n[3] Learning a Multi-View Stereo Machine, NIPS 2017. \n\nIn particular, [3] is very similar as it also fuses 2D features from multiple views on a shared 3D volume which is later projected to 2D views for depth estimation. \n\nThe quality advantage over existing methods is not convincing:\n* Although slightly better than FSM quantitatively in some case, the gap is usualy quite minimal and sometimes non-existence. \n* Qualitative comparisons (e.g. in Fig.6) are not showing any obvious improvements. \n* How do the depth maps look if viewed as point cloud? This can also provide insights if the proposed method has a better agreement between adjacent views. \n\nBecause of the volumetric feature representation, there is a trade-off between resolution and compute. This is not a concern for image-space models. \n\nA few more minor issues:\n* The \"outer product\" operation in Fig.3 is not mentioned anywhere in the text and contradicts the architecture given in the supplementary. \n* The loss terms are not given precise definitions. \n* L.257 (\"provide our attempts to reproduce and discussions in the supplementary material.\") -- not found in supplementary. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept