DeepSFM: Structure From Motion Via Deep Bundle Adjustment
Authors: Xingkui Wei
Yinda Zhang
Zhuwen Li
Yanwei Fu
Xiangyang Xue
Venue:  (2020)
Keywords: Computer Vision
Bundle Ajustment
Structure from Motion

Abstract: Structure from motion (SfM) is an essential computer vision problem which has not been well handled by deep learning. One of the promising trends is to apply explicit structural constraint, e.g. 3D cost volume, into the network. In this work, we design a physical driven architecture, namely DeepSFM, inspired by traditional Bundle Adjustment (BA), which consists of two cost volume based architectures for depth and pose estimation respectively, iteratively running  to improve both. In each cost volume, we encode not only photo-metric consistency across multiple input images, but also geometric consistency to ensure that depths from multiple views agree with each other. The explicit constraints on both depth (structure) and pose (motion), when combined with the learning components, bring the merit from both traditional BA and emerging deep learning technology. Extensive experiments on various datasets show that our model achieves the state-of-the-art performance on both depth and pose estimation with superior robustness against less number of inputs and the noise in initialization.


Under review as a conference paper at ICLR 2020
DEEPSFM: STRUCTURE FROM MOTION VIA DEEP BUNDLE ADJUSTMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Structure from motion (SfM) is essential computer vision problem which has not been well handled by deep learning. One of the promising trends is to apply explicit structural constraint, e.g. 3D cost volume, into the network. In this work, we design a physical driven architecture, namely DeepSFM, inspired by tradi- tional Bundle Adjustment (BA), which consists of two cost volume based archi- tectures for depth and pose estimation respectively, iteratively running to improve both. each cost volume, we encode not only photo-metric consistency across multiple input images, but also geometric consistency to ensure that depths from multiple views agree with each other. The explicit constraints on both depth (strue- ture) and pose (motion), when combined with the learning components, bring the merit from both traditional BA and emerging deep learning technology. Extensive experiments on various datasets show that our model achieves the state-of-the -f--oh--rterrt art performance on both depth and pose estimation with superior robustness against less number of inputs and the noise in initialization.
INTRODUCTION
Structure from motion (SIM) is a fundamental human vision functionality which recovers 3D struc- tures from the projected retinal images of moving objects or scenes. It enables machines to sense and understand with the 3D world and is critical in achieving real-world artificial intelligence. Over decades of researches, there has been lot of great success on SfM; however, the performance is far from perfect.
Conventional SfM approaches (Agarwal et al. 2011 Wu al. 201 la Engel et al. 2017 Delaunoy & Pollefeys 2014) heavily rely on MoundleAAdjstmeen (BA) (Triggs et al., 1999 Agarwal et al.] 2010 in which 3D structures and camera motions each view are jointly optimized via Levenberg- Marquardt (LM) algorithm (Nocedal & Wright 2006) according to the cross-view correspondence. Though successful in certain scenarios, conventional SfM based approaches are fundamentally re- stricted by the coverage of the provided multiple views and the overlaps among them. They also typically fail to reconstruct textureless or non-lambertian (e.g. reflective or transparent) surfaces due to the missing of correspondence across views. As a result, selecting sufficiently good input views and the right scene requires excessive caution and is usually non-trivial to even experienced user.
Recent researches resort deep learning to deal with the typical weakness of conventional SIM. Early effort utilizes deep neural network as a powerful mapping function that directly regresses the structures and motions (Ummenhofer al. 2017 Vjjaynnrraimmhan al. 2017; Zhou al. 2017] Wang et al., 2017). Since the geometric constraints of structures and motions are not explicitly en- forced, the network does not learn the underlying physics and prone to overfitting. Consequently, they do not perform as accurate as conventional SfM approaches and suffer from extremely poor generalization capability. Most recently, the 3D cost volume (Teed Deng. 2018) has been intro- duced to explicit leveraging photo-consistency in differentiable way, which significantly boosts the performance of deep learning based 3D reconstruction. However, the camera motion usually has to be known (Yao et al. 2018 Im et al. 2019) predicted via direct regression (Ummenhofer al. 2017 Zhou et al. 2017) Teed & Deng 2018), which still suffer from generalization issue.
In this paper, explicitly enforce photo-consistency, ceammetrrc-consstteneyy and camera motion constraints in a unified deep learning framework. particular, our network includes depth based cost volume (D-CV) and a pose based cost volume (P-CV). D-CV optimizes per-pixel depth values

---

Under review as a conference paper at ICLR 2020
with the current camera poses, while P-CV optimizes camera poses with the current depth esti- mations. Conventional 3D cost volume enforces photo-consistenge by unprojecting pixels into the discrete camera fronto-parallel planes and computing the photometric (i.e. image feature) difference as the cost. In addition to that, our D-CV further enforces geometric-consistengg among cameras with their current depth estimations by adding the geometric (i.e. depth) difference the cost. Note that the initial depth estimation can be obtained using the conventional 3D cost volume. For pose es- timation, rather than direct regression, our P-CV discretizes around the current camera positions, and also computes the photometric and/or geometric differences by hypothetically moving the camera into the discretized position. Note that the initial camera pose can be obtained by rough estimation from the direct regression methods such (Ummenhofer et al. 2017). Our framework bridges the gap between the conventional and deep learning based SIM by incorporating explicit constraints of photo-consistency, geometric-consistnncy and camera motions all in the deep network.
The closest work in the literature is the recently proposed BA-Net (Tang & Tan 2018), which also aims to explicitly incorporate multi-view geometric constraints in deep learning framework. They achieve this goal by integrating the LM optimization into the network. However, the LM iterations unrolled with few iterations due to the memory and computational inefficiency, and thus it may lead to non-optimal solutions. In contrast, our method does not have a restriction on the number of it- erations and achieves empirically better performance. Furthermore, LM in SfM originally optimizes point and camera positions, and thus direct integration of LMM still requires good correspondences. To evade the correspondence issue in typical SIM, their models employ a direct regressor predict depth at the front end, which heavily relies on prior in the training data. In contrast, our model is fully physical-driven architecture that less suffers from over-fitting issue for both depth and pose estimation.
To demonstrate the superiority of our method, we conduct extensive experiments on DeMoN datasets, ScanNet and ETH3D. The experiments show that our approach outperforms the state-of- the-art [Schonberger & Frahm (2016); Ummenhofer al. (2017); Tang & Tan (2018).
2 RELATED WORK
There is large body of work that focuses on inferring depth or motion from color images, ranging from single view, multiple views and monocular video. We discuss them in the context of our work.
Single-view Depth Estimation. While ill ill-posed, the emerging of deep learning technology enables the estimation of depth from a single color image. The early work directly formulates this into a per-pixel regression problem (Eigen et al.] 2014). and follow-up works improve the performance by introducing multi-scale network architectures (Eigen et al., 2014 Eigen & Fergus 2015), skip- connections (Wang al. 2015 Liu et al.] 2016). powerful decoder and post process (Garg et al.) 2016 Laina et al. 2016; Kuznietsov et al. 2017] Wang et al. 2015 Liu et al. 2016), and new
loss functions (Fu et al. 2018). Even though single view based methods generate plausible results, the models usually resort heavily to the prior in the training data and suffer from generalization capability. Nevertheless, these methods still act important component in some multi-view
systems (Tang & Tan 2018)
Simultaneously estimating 3d structure and camera motion is a well studied problem which has a traditional tool-chain of techniques (Furukawa et al. 2010 New- combe et al. (20TT: Wu et al. (01Tb). Structure from Motion(S(M) has made great progress in many aspects. Lowe 2004); Han
Traditional Prruccurr-from-Mottiee
Han et al. (2015) aim at improving features and (navell (211T) introduce new optimization techniques. More robust structures and data representations are introduced by Gherardi et al. (2010); Schonberger & Frahm (2016). Simultaneous Localization and Sapping(SIAAM) sys- tems track the motion of the camera and build 3D structure from video sequence (Newcombe al, 2011 Engel al. [014]] Mur-Artal al. 2015). Mur-Artal & Tardós 2017). Engel et al. (2014) propose the photometric bundle adjustment algorithm to directly minimize the photometric error of aligned pixels. However, traditional SfM and SLAM methods are sensitive low texture region, occlusions, moving objects and lighting changes, which limit the performance and stability.
Deep Learning for Structure-from-Mootiee Deep neural networks have shown great success in stereo matching and Structure-frot 2017): Vijayanarasimhan et al. (2017): /hoou et al. (007) regress depth map and camera pose di-
problems. |Ummenhoter et al.] (2017); WWang et al.
2

---

Under review as a conference paper at ICLR 2020
Source extraction
Peop based volume
Cost volume 3D
rigression
Initial or estimated
Reference imcage
Feature extraction
Depth basard
volumse
Depth ndinument
Depth regression
Figure I: Overview of our method. 2D CNN is used to extract photometric feature construct cost volumes. Initial source depth maps are used to introduce geometry consistency. A series of 3D CNN layers are applied for both pose based cost volume and D-CV. Then context network and depth regression operation are applied to produce predicted depth map of reference image.
rectly in a supervised manner or by introducing photometric constraints between depth and motion as a self-supervision signal. Such methods solve the camera motion as regression problem, and the relation between camera motion and depth prediction is neglected.
Recently, some methods exploit multi-view photometric or feature-metric constraints to enforce the relationship between dense depth map and the camera pose in network. The SE3 transformer layer is introduced by [Ted & Deng (2018), which uses geometry to map flow and depth into a camera pose update. [Wagg al. (2018 propose the differentiable camera motion estimator based on the Direct Visual Odometry (Steinbrucker
al. 201T). Clark et al. (2018) using LSTM-RNN (Hochreiter al. 2001) the optimizer to solve nonlinear least squares in two-view SfM. Tang & Tan (2018) train a network to generate a set of basis depth maps and optimize depth and camera poses in BA-layer by minimizing feature-metric eπror.
ARCHITECTURE
Our framework receives frames of scene from different viewpoints, and produces photo-metricall and geometrically consistent depth maps across frames and the corresponding camera poses. Similar to BA, also assume initial structures (i.e depth maps) and motions (i.e. camera poses) are given. Note that the initialization not necessary to be super accurate for the good perfor- mance using our framework and thus can be easily obtained from some direct regression methods (Ummenhofer et al. 2017).
Now introduce the detail of model DeepSFM. Without loss of generality, describe model taking two images as input, namely the reference image and the source image, as an example, and all the technical components can be extended for multiple images straightforward. As shown in Figure we first extract feature maps from input images through shared encoder. We then sample the solution space for depth uniformly in the inverse-depth space between predefined minimum and maximum range and camera pose around the initialization respectively. After that, we build cost volumes accordingly to reason the confidence of each hypothesis. This is achieved by validating the consistency between the feature of the reference view and the ones warped from the source image. Besides photo-metric consistency that measures the color image similarity, we also take into account the geometric consistency across warped depth maps. Note that depth and pose require different designs cost volume to efficiently sample the hypothesis space. Gradients can back-propagate through cost volumes, and cost volume construction does not affect any trainable parameters. The cost volumes are then fed into 3D CNN to regress new depth and pose. These updated value can be used to create
cost volumes, and the model improves the prediction iteratively.
For notations, we denote [1,)a as the image sequences one scene, (D,} 1 as the corresponding ground truth depth maps, (K ] the camera intrinsics, (R, 12 the ground truth rotations
3

---

Under review as a conference paper at ICLR 2020
and translations of camera, (D, Hi and {R, 1. 4/2-1 as initial depth maps and camera pose param- eters for constructing cost volumes, where n is the number of image samples.
3.1 2D FEATURE EXTRACTION
Given the input sequences [I, H==1 we extract the 2D CNN feature {F,) for each frame. Firstly, a 7 layers' CNN with kernel size 3 x is applied to extract low contextual information. Then we adopt a spatial pyramid pooling (SPP) (Kaiming et al. al.] (0014) module, which can extract hierarchical multi- scale features through 4 average pooling blocks with different pooling kernel size (4 4, 8 8, 16 x 16, 32 x 32). Finally, we pass the concatenated features through 2D CNNs to get the 32-channel image features after upsampling these multi-scale features into the same resolution. These image sequence features are used by the building of both our depth based and pose based cost volumes.
3.2 DEPTH BASED COST VOLUME (D-CV)
Traditional plane sweep cost volume aims to back-project the source images onto successive virtual planes in the 3D space and measure photo-consistency error among the warped image features and reference image features for each pixel. Different from the cost volume used in previous multi- view and structure- frotmmmoooo methods, we construct D-CV further utilize the local geometric consistency constraints introduced by depth maps. Inspired by the traditional plane sweep cost volumes, our D-CV is concatenation of three components: the reference image features, the warped source image features and the homogeneous depth consistency maps.
Hypothesis Sampling To back-project the features and depth maps from source viewpoint to the 3D space in reference viewpoint, we uniformly sample set of L virtual planes {d/ in the inverse-depth space which are perpendicular to the forward direction (z-axis) of the reference view- point. These planes serve the hypothesis of the output depth map, and the cost volume can be built upon them.
Feature warping. To construct our D-CV, we first warp source image features F, (of size CHannel x Width x Height to each of the hypothetical depth map planes d, using camera intrinsic matrix K and initial camera poses {R;, } according to:
(K (K-"u)en di
F (u) = F, (wx), ih K [R]]C]]
where u and ūn are the homogeneous coordinates of each pixel in the reference view and the pro- jected coordinates onto the corresponding source view. (u(u) denotes the warped feature of the source image through the 1-th virtual depth plane. Note that the projected homogeneous coordinates ün are floating numbers, and we adopt differentiable bilinear interpolation to generate the warped feature map Fa. The pixels with no source view coverage are assigned with zeros. Following Im al.) (2019), we concatenate the reference feature and the warped reference feature together and obtain a 2CH x L x W H 4D feature volume.
addition to photometric consistency, to exploit geometric consistency and promote the quality of depth prediction, add two more channels each virtual plane: the warped initial depth maps from the source view and the depth map of the virtual plane from the perspective of the source view. Note that the former is the same as image feature warping, while the latter requires coordinate transformation from the reference camera to the source camera.
Depth consistency.
In particular, the first channel is computed as follows. The initial depth map of source image first down-sampled and then warped to hypothetical depth planes based on initial camera pose similarly to the image feature warping:
(u) = D (ū))
(2)
where the coordinates u and ûn are defined in Eq. and (u) represents the warped one-channel depth map the 1-th depth plane. One distinction between depth warping and feature warping that we adopt nearest neighbor sampling for depth warping, instead of bilinear interpolation. A comparison between the two methods are provided in Appendix C.
4

---

Under review as a conference paper at ICLR 2020
The second channel contains the depth values of the virtual planes in the reference view by seeing them from the source view. To transform the virtual planes to the source view coordinate system, apply I function each virtual plane di in the following:
(K (K-'u) u) dj
T(d:) ~ [R]]t]]]
(3)
We stack the warped initial depth maps and the transformed depth planes together, and get depth volume of size 2 x L x x H.
By concatenating the feature volume and depth volume together, we obtain 4D cost tensor of size (2CH - 2) x L x W x H. Given the 4D cost volume, our network learns a cost volume of size x W x H using several 3D convolutional layers with kernel size 3 x 3 3. When there is more than one source image, we get the final cost volume by averaging over multiple input source views.
3.3 POSE BASED COST VOLUME (P-CV)
In addition to the construction of D-CV, we also propose a P-CV, aiming at optimizing initial camera poses through both photometric and geometric consistency. Instead of building cost volume based on hypothetical depth map planes, our novel P-CV is constructed based on a set of assumptive camera poses. Similar to D-CV, P-CV is also concatenated by three components: the reference image features, the warped source image features and the homogeneous depth consistency maps. Given initial camera pose parameters {R. ). we uniformly sample a batch of discrete candidate camera poses around. Since jointly sampling camera rotation and translation along 6-DoF is costly, we shift rotation and translation separately by keeping one frozen while sampling the other one. In the end, a group of P virtual camera poses noted as (R } around input pose are obtained for cost volume construction.
The posed-based cost volume is also constructed by concatenating image features and homogeneous depth maps. However, source view features and depth maps are warped based on sampled camera poses. For feature warping, compute üp following equations:
(K 1K--u) D
Up K R
(4)
where D is the initial reference view depth. Similar to D-CV, we get warped source feature map Fi after bilinear sampling and concatenate with reference view feature map. We also transform the initial reference view depth and source view depth into one homogeneous coordinate system, which enhances the geometric consistency between camera pose and multi view depth maps.
After concatenating the above feature maps and depth maps together, volume of size (2CHH + 2) x P x W x H, where W and H are the width and height of feature map, CH is the number of channels. We get output of size 1 P x from the above 4-D tensor after eight 3D convolutional layers with kernel size 3 x 3 x 3, three 3D average pooling layers with stride 1 and one global average pooling at the end.
again build 4D cost
size 2
3.4 COST AGGREGATION AND REGRESSION
For depth prediction, we follow the cost aggregation technique introduced by Im al. (2019). We adopt context network, which takes reference image features and each slice of the coarse cost volume after 3D convolution as input and produce the refined cost slice. The final aggregated depth based volume is obtained by adding coarse and refined cost slices together. The last step to get depth prediction of reference image is depth regression. We pass each slice of D-CV through a soft-max function get the probability of every depth value l. Then the weighted sum of all hypothetical depth values is regarded as predicted depth map; this operation is called soft-argmax. We can also get the predicted coarse depth map by the same way using coarse D-CV. For camera poses prediction, we also apply a soft- aoft-argmax function on pose cost volume and get the estimated output rotation and translation vectors.
5

---

Under review as a conference paper at ICLR 2020
3.5 TRAINING
The DeepSFM learns the feature extractor, cost aggregation, and the regression layers in a supervised way. We denote R, and f; as predicted rotation angles and translation vectors of camera pose. Then the pose loss function is defined as the L1 distance between prediction and groundtruth: Crotation R R and Ltransiation
t: We denote DM and D, predicted coarse depth map and refined depth map for the i-th image, then the depth loss function is defined as following equation:
i
Ldepth EXH(DR D.) H(D;, D.)
(5)
where À is weight parameter and function H is Huber loss.
Our final objective becomes
finna Xy Erotation + d Atrunstattion + dehdh
(6)
We follow two rules to set År, d and Aa: 1) the loss term provides gradient on the same order of numerical value range, such that no single loss term could dominate the training process, since accuracy in depth and camera pose are both important reach good consensus. 2) found in practice the camera rotation has higher impact on the accuracy of the depth but not the opposite. To encourage better performance of pose, set a relatively large A, In practice, the weight parameter À to balance loss objective is set to 0.7, while A = 0.8, X = 0.1 and da = 0.1.
The RGB sequences, corresponding ground-truth depth maps and camera intrinsics and extrinsics are fed as input samples. We initialize the 2D feature extraction layers with pre-trained DPSNet weight. The initial depth maps and camera poses (D and {R, () are obtained from De- MoN. To keep correct scale, we multiply translation vectors and depth maps by the norm of the ground truth camera translation vector. The whole training and testing procedure are performed as four iterations. During each iteration, we take the predicted depth maps and camera poses of previous iteration as new [D]]² and (R,, |/ for cost volume construction.
We implement our system using PyTorch framework. The training procedure takes 6 days on 3 NVIDIA TITAN GPUs on 160k training sequences. The training batch size set to 4, and the Adam optimizer (81 0.9, 32 0.999) is used with learning rate 2 x 10-4 which decreases to x 10-5 after 2 epochs. Within the first two epochs, the parameters in 2D CNN feature extraction module are frozen, and the ground truth depth maps for source images used to construct D-CV and P-CV, which are replaced with predicted depth maps from network latter epochs. During training process, the length of input sequences is 2 (one reference image and one source image). The I, for D-CV is set to 64 and the N for P-CV is 10. The range of both cost volumes is adapted during training and testing.
EXPERIMENTS
4.1 DATASETS
We evaluate DcepSFM on widely used datasets and compare to state-of-the-art methods on accuracy and generalization capability.
DeMoN Datasets Proposed in DeMoN (Ummenhofer et al. 2017), this dataset contains data from various sources, including SUN3D (Xiao et al.J 2013), RGB-D SLAM (Sturm et al. (012). and Scenesl1 (Chang et al. 2015). To test the generalization capability, we also evaluate on MVS (Fuhrmann et al. 2014) dataset but not use for the training. In all four datasets, RGB image sequences and the ground truth depth maps are provided with the camera intrinsics and camera poses. Note that those datasets together provide a diverse set of both indoor and outdoor, synthetic and real-world scenes. Specifically, Scenes consists of synthetic images rendered from random scenes, on which ground truth camera poses and depth are perfect, but objects are lack of reality in scale and semantics. For training and testing, use the same setting DeMoN.
ETH3D Dataset ETH3D dataset provides a variety of indoor and outdoor scenes with high- precision ground truth 3D points captured by laser scanners, which is more solid benchmark

---

Under review as a conference paper at ICLR 2020
Ground truth depth maps are obtained by projecting the point clouds each camera view. Raw images are in high resolution but resized to 810 x 540 pixels for evaluation due to memory constraint. Again, all the models are trained on DeMoN and tested here.
dataset.
MVS
Dopth
Motion
Scenesl1
Depth
Motion
Method
Trans
Method
Ll-inv
sc-inv
Base- Oracle
0.019
0.197
0.105
0
Base-Oracle
0.023
0.618
0.349
Bise-SIFT
0.056
0.309
0.361
21.180
60.516
Base-S1FF
0.051
0.900
1,027
6.179
56.650
Base-FF
0.055
0,308
0.322
4,834
17.252
Base-FF
0.038
0.793
0.776
1,309
19.426
Base Matlah
10.843
32,736
Base-Marlab
0.917
14,639
DeMoN
0.047
0.202
5,156
14.447
DeMoN
0.019
0.315
0.248
8.918
LS-Net
0.051
0.221
0.311
4.653
11.221
LS-Net
0.010
0,410
0.210
4,653
8,210
BANet
3.499
11.238
0,080
0.210
0.130
10.370
Ours
0.021
0.129
0.079
2.824
9.881
Ours
0.007
0.112
0.064
0.403
5.828
RGB-D
Depth
Motion
Sun3D
Depth
Motion
Mathod
1.1-nv
sc-inv
1.1-rel
Rot
Trans
Method
1.1-inv
sc-inv
1.I-rel
Rot
Trans
Basc-Oracle
0.026
0.398
0.36
0
Basc-Oracle
0.020
0.241
0,220
Base-SIl SIFT
0.050
D.577
0,703
12,010
56.021
Base SIFT
0,029
0.290
0.286
7.702
41,825
Base-FF
0.045
4.709
0,029
0.284
0.297
33.301
Base-Matlab
12.813
49.612
Base-Matlab
5,920
32.298
DeMoN
0.130
0.212
2.641
20.585
DeMoN
0.019
0.114
0.2
1.801
LS-Net
0.019
0,090
0.301
1.010
22.100
LS-Net
0.015
0.189
0.650
1.521
14.347
BANet
0.008
0.087
0.050
2,459
14.900
BANet
0.015
0.110
0,060
1.729
13.260
Ours
0.011
0.071
0.126
1,862
14.570
Ours
0.013
0.093
0.072
1.704
13.107
Table I: Results on MVS, SUN3D, RGBD and Scenesl I, the best results are noted by Bold.
4.2 EVALUATION
DeMoN Datasets Our results on DeMoN datasets and the comparison other methods are shown in T We cite results of some strong baseline methods from DeMoN paper, named as Base- -racle,,,,,, Base-SIFT, Base-FF and Base-Matlab respectively (Ummenhofer et al. (017). Base-Oracle estimate depth with the ground truth camera motion using SGM (Hirschmuller 2005). Base-SIFT, Base- FF and Basc-Matlab solve camera motion and depth using feature, optical flow, and KLT tracking correspondence from 8-pt algorithm (Hartley 1997). We also compare to some most recent state- of-the-art methods LS-Net (Clark et al. 2018)) and BA-Net (Tang & Tan 2018). LS-Net introduces the learned LSTM-RNN optimizer minimizing photometric error for stereo reconstruction. BA- Net is the most recent work that minimizes the feature-metric error between multi-view via the differentiable Gevenberg-Marquaar (Lourakis & Argyros 2005) algorithm.
To make a fair comparison, we adopt the same error metrics as DeMoN for depth and camera pose evaluation. Ll-iny computes the disparity map errors, and sc-inv is scale-invariant error metric.
Error metric
Accuracy metric(Ō < a')
Method
abs rel
abs diff
sq rel
log rms
a²
a
COLMAP
0.324
0.615
36.71
2.370
0.349
86.5
90.3
92.7
DeMoN
0.191
0.726
0.365
1.059
0.240
73.3
89.8
95.1
Ours 0.127
0.661
0.278
1.003
0.195
84.1
93.8
96.9
Table 2: Results on ETH3D (Bold: best; a = 1.25). abs abs_rel, abs_difff, sq_rel, rms, and log log_rms, are absolute relative error, absolute difference, square relative difference, root mean square and log root mean square, respectively.
7

---

Under review as a conference paper at ICLR 2020
LI-rel measures the depth errors relative to the ground truth depth, which emphasize depth estima- tion of close range in the scene. For camera poses evaluation, the angles between the prediction and the ground truth rotation and translation are shown as Rot and Trans respectively.
Our method outperforms all traditional bascline methods and DeMoN on both depth and camera poses. When compared to more recent LS-Net and BA-Net, our method produces better results in most metrics of the four datasets. On RGB-D dataset, our performance is comparable to the state- of-the-art due relatively higher noise in the RGB-D ground truth. LS-Net trains an initialization network which regresses depth and motion directly before adding the LSTM-RNN optimizer. The performance of the RNN optimizer is highly affected by the accuracy of the regressed initializa- tion. The depth results of LS-Net are consistently poorer than BA-Net and our method, despite better rotation parameters are estimated by L-S-Net on RGB-D and Sun3D datasets with very good initialization. Our method is slightly inferior to BA-Net on the L1-rel metric, which is probably due to that we sample 64 virtual planes uniformly as the hypothetical depth set, while BA-Net op- timizes depth prediction based on set of 128-channel estimated basis depth maps that are more memory consuming but have more fine-grained results empirically. Despite all that, it is shown that our learned cost volumes with geometric consistency work better than the photometric bundle ad- justment (e.g. used in BA-Net) in most scenes. In particular, we improve mostly on the Scenes dataset, where the ground truth is perfect but the input images contain a lot of texture-less regions, which are challenging to photo-consistency based methods.
ETH3D We further test the generalization capability ETH3D. We provide comparisons to COLMAP (Schonberger & Frahm 2016) and DeMoN on ETH3D. COLMAP is state-of-the-art Aimuctre-from-Mootii method, while DeMoN introduces classical deep network architecture that directly regress depth and motion in a supervised manner. In the accuracy metric, the error s de- fined max( 1. ). and the thresholds typically as [1.25, 1.25², 1. 1.25%). In Table 2 yn y."
method shows the best performance overall among all the comparison methods. Our method pro- duces better results than DeMoN consistently, since impose geometric and physical constraints onto network rather than learning to regress directly. When compared with COLMAP, our method performs better most metrics. COLMAP behaves well in the accuracy metric (i.e. abs diff). However, the presence of outliers is often observed in the predictions of COLMAP, which leads to poor performance in other metrics such abs_rd rel and sq_rel, since those metries are sensitive to outliers. We put more qualitative comparisons with COLMAP in Appendix C. For more comparison on generalization, another experiment on ScanNet is provided in Appendix B
4.3 MODEL ANALYSIS
In this section, we analyze our model on several aspects to verify the optimality and show advantages over previous methods.
oursais.re (R
0.22
base-aa
-.24
2.2
0.23
0.24
2.0
0.22
8.2
16
0.18
9.20
6.12
018 10
(a) depth metrics comparison
(b) camera pose metrics comparison
Figure 2: Comparison with baseline during iterations. Our work converges at better position. (a) abs relative error and log RMSE (b) rotation and translation degree error.
Iterative Improvement Our model lines) shows our performance over iterations when initialized with the prediction from DeMoN. As can be seen, our model effectively reduces both depth and pose errors upon the DeMoN output.
iteratively to reduce the prediction error. Figure (solid
8

---

Under review as a conference paper at ICLR 2020
Throughout the iterations, better depth and pose benefit each other by building accurate cost volume, and both are consistently improved. The whole process is similar to coordinate descent algorithm, and finally converges at iteration 4.
Effeet of P-CV We compare DeepSFM to a baseline method for our P-CV. this baseline, the depth prediction is the same DeepSFM, but the pose prediction network replaced by direct visual odometry model SSteinbrücker et al. (2011). which updates camera parameters by minimizing pixel-wise photometric error between image features. Both methods are initialized with DeMoN results. As provided in Figure 2 DeepSFM consistently produces lower errors on both depth and pose over all the iterations. This shows that our P-CV predicts more accurate pose and performs more robust against noise depth at early stages.
0.3
0.5
0.38
RGB Image
GT
2 view
0.8
0.
3 view
view
6 vieww
Peites
(a) Abs relative error metric compari-
(b) Our results with different view numbers
Figure 3: Depth map results w.r.t the number images.
View Number DeepSFM works still reasonably well with fewer views due to the free from opti- mization based components. To show this, compare to COLMAP with respect to the number of input views on ETH3D. As depicted in Figure 3 more images yield better results for both meth- ods expected. However, our performance drops significantly slower than COLMAP with fewer number of inputs. Numerically, DeepSFM cuts the depth error by half under the same number of views as COLMAP, or achieves similar error with half number of views required by COLMAP, This clearly demonstrates that DeepSFM is more robust when fewer inputs are available.
CONCLUSIONS
We present a deep learning framework for Structurefromm-Motion, which explicitly enforces photo- metric consistency, geometric consistency and camera motion constraints all in the deep network. This is achieved by two key components namely D-CV and P-CV. Both cost volumes measure the photo-metric errors and geometric errors but hypothetically move reconstructed scene points (structure) or camera (motion) respectively. Our deep network can be considered as enhanced learning based BA algorithm, which takes the best benefits from both learnable priors and geometric rules. Consequently, our method outperforms conventional BA and state-of-the-art deep learning based methods for SIM.
REFERENCES
Sameer Agarwal, Noah Snavely, Steven M Seitz, and Richard Szeliski. Bundle adjustment in the large. In European conference computer vision, pp. 29 -2.4.. Springer, 2010.
Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome in day. Communications of the ACM, 54(10):105-112 201
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information 3d model repository. arXiv preprint IrXiv: 2015.

---

Under review as a conference paper at ICLR 2020
Ronald Clark, Michael Bloesch, Jan Czarnowski, Stefan Leutenegger, and Andrew J Davison. Learning to solve nonlinear least squares for monocular stereo. Proceedings of the European Conference Computer Vision (ECCV), pp. 284 284-299, 2018.
Amaäl Delaunoy and Mare Pollefeys. Photometric bundle adjustment for dense multi-view 3d mod- cling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 1486-1493, 2014.
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with common multi-scale convolutional architecture. In The JEEE International Conference on Computer Vision (ICCV), December 2015.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Advances neural information processing systems, pp. 2366-2374, 2014.
Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In European conference on computer vision, 834-849, Springer, 2014.
Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct sparse odometry. IEEE transactions on pattern analysis and machine intelligence, 40(3):611-625, 2017.
Huan Fu, Mingming Gong. Chaohui Wang, Kayhan Batmanghelich. and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Simon Fuhrmann, Fabian Langguth, and Michael Goesele. Mve-a multi-view reconstruction envi- ronment. In GCH, pp. 11-18, 2014.
Yasutaka Furukawa, Brian Curless, Steven M Seitz, and Richard Szeliski. Towards intereet-scaec multi-view stereo. 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 1434 1441. IEEE, 2010.
Ravi Garg, Vijay Kumar BG, Gustavo Carneiro, and Ian Reid. Unsupervised enn for single view depth estimation: Geometry to the rescue. In European Conference on Computer Vision (ECCV), pp. 740-756. Springer, 2016.
Riccardo Gherardi, Michela Farenzena, and Andrea Fusiello. Improving the efficiency of hierarchi- In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1594-1600. IEEE, 2010.
structure-and-ne
Xufeng Han, Thomas Leung, Yangqing Jia, Rahul Sukthankar, and Alexander Berg. Matchnet: Unifying feature and metric learning for patch-based matching, In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3279-3286, 2015.
Richard I Hartley. In defense of the eight-point algorithm. IEEE Transactions on pattern analysis and machine intelligence, 1906)5500-593, 1997.
Heiko Hirschmuller. Accurate and efficient stereo processing by semi-global matching and mu- tual information. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'OS) '05), volume 2, pp. 807-814. IEEE, 2005.
Sepp Hochreiter, A Steven Younger, and Peter Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87-94. Springer, 2001.
Sunghoon Im, Hae-Gon Jeon, Stephen Lin, and In So Kweon. Dpsnet: End-to-end deep plane sweep stereo. In International Conference Learning Representations, 2019. URL https:
He Kaiming, Zhang Xiangyu, Ren Shaoqing, and Jian Sun. Spatial pyramid pooling in deep convo- lutional networks for visual recognition. In European Conference on Computer Vision (ECCV), 2014.
10

---

Under review as a conference paper at ICLR 2020
Yevhen Kuznietsov, Jorg Stuckler, and Bastian Leibe. Semi- supervsuee deep learning for monocular depth map prediction. In The IEEE Conference on Computer Vision and Pattern Recognition Jull 2017.
Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 2016 Fourth International Con- ference on 3D Vision (3DV), pp. 239-248. IEEE, 2016.
Fayao Liu, Chunhua Shen, Guosheng Lin, and lan Reid. Learning depth from single monocular images using deep convolutional neural fields. IEEE transactions on pattern analysis and machine intelligence, 38( 28(10):0024-0339 2016.
MLA Lourakis and Antonis A Argyros.
beenbbrr-marquartt the most efficient optimization algorithm for implementing bundle adjustment? In Tenth IEEE International Conference Computer Vision (ICCV'05) Volume I, volume 2. pp. 1526-1531. IEEE, 2005.
David G Lowe. Distinctive image features from scale- inver-in keypoints. International journal of computer vision, 60(2):91-110, 2004.
Raul Mur-Artal and Juan Tardós. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Transactions on Robotics, 3355):225--1262 2017.
Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: versatile and accu-
rate monocular slam system. IEEE transactions on robotics, 3165):1147-1163, 2015.
Richard A Newcombe, Steven J Lovegrove, and Andrew Davison. Dtam: Dense tracking and mapping in real-time. In 2011 international conference computer vision, pp. 2320-2322 IEEE, 2011.
Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
Johannes I. Schonberger and Jan-Michael Frahm. Structure-from-motde revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. pp. 41.41044111 2016.
Noah Snavely. Scene reconstruction and visualization from internet photo collections: A survey.
IPSJ Transactions on Computer Vision and Applications, 3:44-66, 2011.
Frank Steinbrücker, Jürgen Sturm, and Daniel Cremers. Real-time visual odometry from dense rgb-d images. In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 719-722, IEBE, 2011.
Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A bench- mark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 573-580. IEEE, 2012.
Chengzhou Tang and Ping Tan. Ba-net: Dense bundle adjustment network.
arXiv preprint
N80::18006.4807,, 2018
Zachary Teed and Jia Deng. Deepy2d: Video to depth with differentiable structure from motion. arXiv preprint arXiv: 10f2.0480000000 2018
Bill Triggs, Philip F McLauchlan, Richard Hartley, and Andrew W Fitzgibbon. Bundle ad- justment-a modern synthesis. In International workshop on vision algorithms, pp. 298-372.
Springer, 1999.
Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Doso- vitskiy, and Thomas Brox. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5038- 5047, 2017.
Sudheendra Vijayanarasimhan, Susanna Ricco, Cordelia Schmid, Rahul Sukthankar, and Kate- rina Fragkiadaki. Sfm-net: Learning of structure and motion from video. arXiv preprint arXiv: 1704. 07804, 2017.
11

---

Under review as a conference paper at ICLR 2020
Chaoyang Wang, José Miguel Buenaposada, Rui Zhu, and Simon Lucey. Learning depth from monocular videos using direct methods. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1022-2030, 2018.
Peng Wang, Xiaohui Shen, Zhe Lin, Scott Cohen, Brian Price, and Alan L. Yuille. Towards unified depth and semantic prediction from a single image. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
Sen Wang. Ronald Clark, Hongkai Wen, and Niki Trigoni. Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks. In 2017 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 2043-2050. IEEE, 2017.
Changchang Wu, Sameer Agarwal, Brian Curless, and Steven M Seitz. Multicore bundle adjustment. In CVPR 201 pp. 3057-3064. IEEE, 201 la.
Changchang Wu et al. Visualsfm: A visual structure from motion system. 2011b.
Jianxiong Xiao, Andrew Owens, and Antonio Torralba. Sun3d: database of big spaces recon- structed using sfm and object labels. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1625-1632, 2013.
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstruc- tured multi-view stereo. In Proceedings of the European Conference Computer Vision (ECCV), pp. 767-783, 2018.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1851-1858, 2017.
12

---

Under review as a conference paper at ICLR 2020
3x3 2D conv, 32 channels, stride=2, dilation=1 batch norm. ReLU
3x3 2D conv, 32 channels, stride=l, dilation=1 batch norm. ReLU
3x3 2D conv, 32 channels, stride=1, dilation=1 batch norm. ReLU
3x3 2D conv, 32 channels, stride=1, dilation=1 batch norm. ReLU
3x3 2D conv, 64 channels, stride=2, dilation=1 batch norm. ReLU
3x3 2D conv, 128 channels, stride=1, dilation=1 batch norm. ReL.U
3x3 2D conv, 128 channels, stride=1, dilation=2 batch norm. ReLU
32x32 AvgPool 3x3 2D conv, 32 channels, stride=1 batch norn. ReLU
32x32 AvgPool
8x8 AvgPool 3x3 2D conv, 32 channels, stride=1, batch norm. ReLU
4x4 AvgPool 3x3 2D conv, 32 channels, stride=1, batch norm. ReLU
3x3 2D conv, 32 channels, stride=1, batch norm. ReLU
3x3 2D conv, 128 channels, stride=1, dilation=1 batch norm. ReLU
1x1 2D conv, 32 channels, stride=1, dilation=1
Figure 4: Detail architecture of feature extractor.
A IMPLEMENTATION DETAILS
Feature extraction module As shown in Figure we build our feature extraction module refer- ring DPSNet (Im al. 2019). The module takes 4W x 4H 3 images as input and output feature maps of size W x H x 32, which are used to build D-CV and P-CV.
Cost volumes Figure 5 shows the detailed components for the P-CV and D-CV. Each channel of cost volume is composed of four components: reference view feature maps, warped source view feature maps, the warped source view initial depth map and the projected reference view depth plane or initial depth map. For P-CV construction, we take each sampled hypothetical camera pose, and carry out the warping process on source view depth maps and initial depth map based on the camera pose. And the initial reference view depth map is projected to align numeric values with the warped source view depth map. Finally those four components are concatenated as one channel of 4D P-CV. We do this on all P sampled camera poses, and get the P channel P-CV. The building approach for D-CV is similar, we take each sampled hypothetical depth plane, and carry out warping process on source view feature maps and the initial depth map. And the depth plane is projected to align with the source view depth map. After concatenation, one channel in D-CV is got. Same computation is done based on all I. virtual depth planes, and the I. channel D-CV is built up.
13

---

Under review as a conference paper at ICLR 2020
Source view feature maps
Initial depth
Fa(u)) F/(R)
By(0 220))
project
concalenate
Virtual planes or initial depth map
Reference view feature maps
Figure 5: Four components in D-CV or P-CV.
Method
Depth
Motion
abs rel
sq
rel
rms
log
inv
Rot
Trans
Ours
0.227
0.170
0.479
0.271
0.268
1.588
30.613
BA-Net
0.238
0.176
0.488
0,279
0.276
1.587
31.005
DeMoN
0.231
0.520
0.761
0,289
0.284
3.791
31.626
LSD-SLAM
0.268
0.427
0.788
0.330
0.323
4.409
34.360
Geometric BA
0.382
1.163
0.876
0.366
0.357
8.560
39.392
Table 3: Results on ScanNet. (sc inv: scale invariant log rms; Bold: best.)
3D convolutional layers The detail architecture of 3D convolutional layers after D-CV is almost the same as DPSNet (Im et al. 2019). except for the fist convolution layer. In order to compati- ble with the newly introduced depth consistent components D-CV, We adjust the input channel number to 66 instead of 64. As shown in Figure 6 for 3D convolutional layers after P-CV, the ar- chitecture is similar to D-CV 3D convolution layers with three extra 3D average pooling layers and finally there is one global average pooling in the dimensions of image width and height, after which we get x tensor.
B EVALUATION ON SCANNET
ScanNet provides large set of indoor sequences with camera poses and depth maps captured from commodity RGBD sensor. Following BA-Net, we leverage this dataset evaluate generalization capability by training models on DeMoN and testing here. The testing set is the same BA-Net, which takes 2000 pairs filtered from 100 sequences.
We evaluate the generalization capability of DeepSFM on ScanNet. Table 3 shows the quantitative evaluation results for models trained on DeMoN. The results of BA-Net, DeMoN, LSD-SLAM and Geometric BA obtained from Tang & Tan (2018). As can be outperforms all previous work, which indicates that our model generalizes well to general indoor environments.
method significantly
14

---

Under review as a conference paper at ICLR 2020
3x3x33D conv, 32 channels, stride=1 batch norm, ReLU 1x2x2 3D AvgPool
3x3x3 3D conv, 32 channels, stride=1 batch norm. ReLU
3x3x3 3D conv, 32 channels, stride=1 batch norm. ReLU 3x3x3 3D conv, channels, stride=1
3x3x3 3D conv, 32 channels, stride=1 batch norm. ReLU 3x3x3 3D conv, channels, stride=1 1x2x2 3D AvgPool
3x3x3 3D conv, 32 channels, stride=1 batch norm. ReLU 3x3x3 3D conv, 32 channels, stride=1
3x3x3 3D conv, 32 channels, stride=1 batch norm. ReLU 3x3x3 3D conv, 32 channels, stride=1 1x2x2 3D AvgPool
3x3x3 3D conv, channels, stride11 batch norm, ReLU Global 3D AvgPool FC, channels
Figure 6: 3D convolutional layers After P-CV.
Initialization
Iteration
Iteration 4
Iteration 6
Iteration 10
Iteration 20
abs relative
0.254
0.153
0.126
0.121
0.120
0.120
log rms
0.248
0.195
0.191
0.190
0.190
0.191
translation
15.20
9.75
9.73
9.73
9.73
9.73
rotation
2.38
1.43
1.40
1.39
1.39
1.39
Table 4: The performance of the optimization iterations for testing.
C SUPPLEMENTAL A ABLATION STUDY
More Iterations for Testing. We take up to four iterations when train DeepSFM. During in- ference, the predicted depth maps and camera poses of previous iteration are taken as initialization of next iteration. To show how DeepSFM performs with more iterations than is trained with, We tested with up to 20 iterations, and it converges at the 6-th iteration.
show results Table 4
Bilinear Interpolation vs Nearest Neighbor Sampling. For the construction of D-CV and P-CV, depth maps are warped via the nearest neighbor sampling instead of bilinear interpolation. Due to the discontinuity of the depth values in depth maps, the bilinear interpolation may bring some side
15

---

Under review as a conference paper at ICLR 2020
MVS Dataset
L1-inv
sc-inv
L1-rel
Rot
Trans
Billinear interpolation
0.023
0.134
0.079
2.867
9.910
Nearest neighbor
0.021
0.129
0.076
2.824
9.881
Table 5: The performance with different warping methods.
effects. It may do damage to the geometry consistency and smooth the depth boundaries. As comparison, we replace the nearest neighbor sampling with the bilinear interpolation. As shown in Ta the performance of our model gains slight drop with the bilinear interpolation, which indicates that the nearest neighbor sampling method is indeed more geometrically meaningful for depth. In contrast, the differentiable bilinear interpolation required for the warping of image features, whose gradients are back propagated to feature extractor layers. Further exploration will be interesting future work.
D VISUALIZATION
We show some qualitative comparison with the previous methods. Since there are source code available for BA-Net (Tang & Tan 2018), we compare the visualization results of our method with DeMoN (Ummenhofer the predicted dense depth map by our method and DeMoN on the DeMoN datasets. As we can see, demon often miss some details in the scene, such as plants, keyboard and table legs. In contrast, our method reconstructs more shape details. FigureB]shows some estimated results from COLMAP and our method on the ETH3D dataset. As shown in the figure, the outputs from COLMAP are often incomplete, especially in textureless area. On the other hand, our method performs better and always produce an integral depth map. In Figure more qualitative comparisons with COLMAP on challenging materials are provided.
(017) and COLMAP (Schonberger & Frahm 2016). Figure 7 shows
16

---

Under review as a conference paper at ICLR 2020
(a)
(b)
(c)
(d)
(e)
Reference images
Ground truth
DeMoN
Ours
Figure Qualitative Comparisons with DeMoN (Ummenhofer al. 2017) on DeMoN datasets.
17

---

Under review as a conference paper at ICLR 2020
(a)
(b)
(c)
(d)
(I)
(f)
Reference images
Ground truth
COLMAP
Ours
Figure 8: Qualitative Comparisons with COLMAP (Schonberger & Frahm 2016 on ETH3D datasets.
18

---

Under review as a conference paper at ICLR 2020
(a)
(b)
(c)
(d)
Reference
images
Ground truth
COLMAP
Ours
Figure 9: Qualitative Comparisons with COLMAP (Schonberger & Frahm 2016) challenging materials. a) Textureless ground and wall. b) Poor illumination scene. c) Reflective and transparent glass wall. d) Reflective and textureless wall.
19


Reviews:
[{"review_id": "rJgQN2PcFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tackles Structure from Motion, one of the canonical problems in computer vision, and proposes an approach that brings together geometry and physics on one hand and deep networks on the other hand. Camera unprojection and warping (of depth maps and features) are used to build a cost volume onto hypothetical planes perpendicular to the camera axis. Similarly, various camera poses are sampled around an initial guess. A deep network regresses form the cost volume to a camera pose and a depth map. The method can be applied iteratively, using the outputs of the current stage as the initial guess of the next one. Training is supervised, and the the results are evaluated on multiple datasets.\n\nI am inclined to recommend accepting the paper for publication, because it addresses a canonical problem, outperforms the state of the art on multiple datasets and brings together geometry / physics and deep learning, which is IMO very a promising and underexplored direction.\n\nI found the method section a bit difficult to read though, and even after several readings I cannot get my head around it. Specifically, here are some issues that I hope the Authors could clarify.\n\n1. In Sec. 3 the Authors write \"We then sample the solution space for depth and pose respectively around their initialization\". However in Sec 3.2 they write \"we uniformly sample a set of L virtual planes {dl} Ll=1 in the inverse-depth space\". In what way are the planes \"around their initialization\"? If the initial depth map spans over multiple orders of magnitude, will the planes be uniformly sampled between the minimum and maximum disparity of the initial map? If yes, it seems that the initial depth map is not really needed, just its minimum and maximum value is needed, but then how come the method can be applied iteratively with respect to depth?\n\n2. The Authors mention that depth maps are warped onto the virtual planes using differentiable bilinear interpolation. Is there a mechanism to protect from interpolating across discontinuities? If no, were bleeding edge artifacts observed?\n\n3. In the introduction, the Authors point that prior methods have trouble dealing with textureless, reflective or transparent approaches, but it's not clear form the paper where it addresses these cases, and if yes, what is the mechanism for that.\n\nLastly, if the authors are not planning to release the code, the implementation details section is a bit too high-level and does not contain enough details to reimplement the Author's technique. For example, \"our network learns a cost volume of size L × W × H using several 3D convolutional layers with kernel size 3 × 3 × 3\"  - more details about this network are needed, as well as the others in the paper.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Syx181Q7cB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a physical driven architecture of DeepSFM to infer the structures from motion. Extensive experiments on various datasets show that the model achieves the state-of-the-art performance on both depth and pose estimation. In general, the paper is clearly written but I still have several concerns. \n1.\tThe paper is easy to follow but the authors are expected to clarify the rationality in integration of the loss function. How the parameter of \\lambda_r, \\lambda_t, and \\lambda_r influence the performance. It would be better if the authors could present some analysis. \n2.\tThe experiments are rather insufficient. The authors are expected to make more comprehensive analysis with the state-of-the-art methods, and also analyze why some alternative methods outperforms the proposed methods in table I and table II. \n3.\tThe experiments in section 4.3 are also expected to be improved. It is difficult to draw a conclusion that the method is better than other ones based on such limited experiments.  \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1e7O15zjr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors propose a SfM model which integrates geometric consistency with a learned pose and depth network. An initial estimate of depth and pose are used to construct pose and depth cost volumes, which are then fed into a pose regression and depth refinement network, to produce a new set of cost volumes, and so on. In this manner, the pose and depth estimation are improved iteratively.\n\nStrengths:\n\nThe proposed model is well motivated and shows strong performance and generalization ability on several datasets. There are convincing experiments to show the importance of the P-CV network.\n\nWeaknesses:\n\nThe authors claim that the LM optimization in BA-Net is memory inefficient and may lead to non-optimal solutions. It’s not clear to me that the proposed method can guarantee optimality any better. It’s also unclear if the proposed method is more memory efficient, since the authors only unroll 4 iterations of it.\n\nOther comments:\n\nIt would be very interesting to see the test time behavior of the network when it is run with more iterations than it is trained with (say 10 or 20), especially since the depth error does not seem to have stopped decreasing at only 4 iterations.\n\nIt’s not made entirely clear whether the training backpropagates through the update/construction of the pose and depth cost volumes. \n\nIn equation 5, “x” should be “i”.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content:  Physical driven architecture of DeepSFM to infer the structures from motion
Discussion:
reviewer 1: well-motivated model with good solid experimental results. not clear about the LM optimization in BA-Net is memory inefficient 
reviewer 2: main issue is the experiments could be improved.
reviewer 3: well written but again experimental section is lacking
Recommendation: Good paper and results, but all 3 reviewers agree experiments could be improved. Rejection is recommended.
Decision: Reject