DeepV2D: Video to Depth with Differentiable Structure from Motion
Authors: Zachary Teed
Jia Deng
Venue:  (2020)
Keywords: Structure-from-Motion
Video to Depth
Dense Depth Estimation

Abstract: We propose DeepV2D, an end-to-end deep learning architecture for predicting depth from video.  DeepV2D combines the representation ability of neural networks with the geometric principles governing image formation. We compose a collection of classical geometric algorithms, which are converted into trainable modules and combined into an end-to-end differentiable architecture. DeepV2D interleaves two stages: motion estimation and depth estimation. During inference, motion and depth estimation are alternated and converge to accurate depth. 

Published as a conference paper at ICLR 2020
DEEPV2D: VIDEO TO DEPTH WITH DIFFERENTIABLE STRUCTURE FROM MOTION
Zachary Teed
Jia Deng
Princeton University zteedlcs www.ppriinnnnnnmmmmmmm
Princeton University
j aidengecs. princeten.edi
ABSTRACT
We propose Vee an end-to-end deep learning architecture for predicting depth from video. DeepV2D combines the representation ability of neural net- works with the geometric principles governing image formation. We compose a collection of classical geometric algorithms, which are converted into trainable modules and combined into an end-to-end differentiable architecture. DeepV2D interleaves two stages: motion estimation and depth estimation. During inference, motion and depth estimation are alternated and converge to accurate depth. Code is available https:://qite
com
INTRODUCTION
In video to depth, the task is to estimate depth from video sequence. The problem has traditionally been approached using Structure from Motion (SfM), which takes collection of images input, and jointly optimizes over 3D structure and camera motion (Schonberger & Frahm (016b). The resulting camera parameter estimates can be used as input to Multi- Stereo in order build more complete 3D representation such as surface meshes and depth maps (Furukawa et al. 2015 Furukawa & Ponce 2010).
In parallel, deep learning has been highly successful in number of 3D reconstruction tasks, In particular, given ground truth depth, a network can learn to predict depth from a single image (Eigen
et al.
2014 Eigen & Fergus. 2015 Laina et al. 2016), stereo images (Kendall et al.) 2017 Mayer
ct al.
2016a). or collections of frames (Zhou et al., [0088 Kar et al., 2017] Tang Tan (018)] Yao
al.
2018). One advantage of deep networks that they can single-image cues such as tex-
ture gradients and shading as shown by their strong performance on depth estimation from a single image (Eigen et al. 2014 Eigen Fergus 2015 Laina et al. 2016). Furthermore, differentiable network modules can be composed so that entire pipelines (i.c. feature extraction, feature matching, regularization) can be learned directly from training data. On the other hand, as recent work has shown, it is often hard to train generic network layers to directly utilize multiview geometry (e.g. using interframe correspondence edge of multiview geometry through specially designed layers or losses (Ummenhofer et al. 2017
recover depth), and it is often advantageous
embed knowl-
Kendall Cipolla 2017 Zhou al. 2017 Vjjaanarasimman et al. 2017 2017: et 2018
In this work, we continue the direction set forth by recent works (Ummenhofer et al., 2017 Kendall
iet al. 22177] Tang & Tan 2018; Zhou et al., 2018: Kar et al. 2017; Wang et al. 2018)) that combine the representation ability of neural networks with the geometric principles underlying image forma-
Depth
Depth
Depth
Motion
Video
Depth
Figure 1: DeD predicts depth from video. It is the composition of classical geometric algo- rithms, made differentiable, and combined into an end-to-end trainable network architecture. Video to depth is broken down into the subproblems of motion estimation and depth estimation, which are solved by the Motion Module and Depth Module respectively.

---

Published as a conference paper at ICLR 2020
tion. We propose V2D, composition of classical geometrical algorithms which turn into differentiable network modules and combine into an end-to-end trainable architecture. Deep in- terleaves two stages: camera motion estimation and depth estimation (Figure TT. The motion module takes depth as input, and outputs an incremental update to camera motion. The depth module takes camera motion input, and performs stereo reconstruction predict depth. At test time, DeepV2I> acts as block coordinate descent, alternating between updating depth and camera motion.
To estimate camera motion we introduce Flow-SE3, new motion estimation architecture, which outputs an incremental update to camera motion. Flow-SE3 takes depth as input, and estimates dense 2D correspondence between pairs of frames. We unroll a single iteration of Perspective- updates over SE3
n-Point (PnP) (Lepetit et al.] 2009 Li et al.] (2022) performing Gauss- perturbations to minimize geometric reprojection error. The new estimate of camera motion can then be fed back into Flow-SE3, which re-estimates correspondence for a finer grain pose update.
Our Depth Module builds upon prior work (Kendall et al., 2017] Yao et al. 2018) and formulates multiview-stereo (MVS) reconstruction as single feed-forward network. Like classical MVS, we leverage geometry to build a cost volume over video frames, but use trainable network for both feature extraction and matching.
Our work shares similarities with prior works (Ummenhofer et al. 2017 Kendall et al. a 2017 Tang & Tan 2018: Zhou et al. 2018 Kar et al.) 2017 Wang et 2018)) that also combine deep learning and multiviets geometry, but is novel and unique in that it essentially "differentialies" a classical SIM pipeline that alternates between stereopsis, dense 2D feature matching, and PnP. As a comparison, DeMon (Ummenhofer et al.) 2017) and Dee (Zhou et al. 2018) differentialize stereopsis and feature matching, but not PnP because they use generic network to predict camera motion.
Another comparison is with BA-Net (Tang & Tan 2018), whose classical analogue is performing bundle adjustment from scratch optimize feature alignment over camera motion and the coef- ficients of limited set of depth maps (depth basis). In other words, BA-Net performs one joint nonlinear optimization over all variables, whereas we decompose the joint optimization into more tractable subproblems and do block coordinate descent. Our decomposition is more expressive in terms of reconstruction since we can optimize directly over per-pixel depth and are not constrained by a depth basis, which can potentially limit the accuracy of the final depth.
In our experiments, we demonstrate the effectiivnesss of Deep across a variety of datasets and tasks, and outperform strong methods such as DeepTAM (Zhou et al., 2018). DeMoN (Ummenhofer jet al.] 2017), BANet (Tang & Tan 2018), and MVSNet (Yao et al.) 2018). As we show, alternating depth and motion estimation quickly converges to good solutions. On all datasets outperform all existing single-view and multi-vien approaches. We also show superior cross-dataset generalizabil- ity, and can outperform existing methods even when training on entirely different datasets.
2 RELATED WORK
Structure from Motion:
Beginning with carly systems designed for small image collections LLoguuet-Higgins 1981 Mohr et al., 1995). Structure from Motion (SfM) has improved dramat- ically in regards to robustness, accuracy, and scalability. Advances have come from improved fea- tures (Lowe) 2004 Han et al. 2015). optimization techniques (Snavely 2009). and more scalable data structures and representations (Schonberger & Frahm 2016a Gherardi
Gherrirr et al. 2010), culminat- ing in number of robust systems capable of large-scale reconstruction task (Schonberger & Frahm 2016a) Snavely 2011 Wu et al., 2011). Ranftl et al. (2016) showed that SfM could be extended to reconstruct scenes containing many dynamically moving objects. However, SfM is limited by the accuracy and availability of correspondence. In low texture regions, occlusions, or lighting changes SfM can produce noisy or missing reconstructions.
Simultaneous Localization and Mapping (SLAM) jointly estimates camera motion and 3D structure from a video sequence (Engel et al. 2014 Mur-Artal et al. 2015 Mur- Artal & Tardos 2017 7 New- combe et al. 2011: Engel et al. 2018). LSD-SLAM (Engel et al. 2014) is unique in that it relies on a featureless approach to 3D reconstruction, directly estimating depth maps and camera pose by minimizing photometric error. Our Motion Network behaves similarly to the tracking component in LSD-SLAM, but we use a network which predicts misalignment directly instead of using intensity gradients. We end up with an easier optimization problem characteristic of indirect methods (Mur-
2

---

Published as a conference paper at ICLR 2020
Artal et al. 2013). while retaining the flexibility of direct methods in modeling edges and smooth intensity changes Engel et al. 2018).
Geometry and Deep Learning: Geometric principles has motivated the design of many deep learning architectures. In video to depth, we need to solve two subproblems: depth estimation and motion estimation.
Depth: End-to-end networks can be trained to predict accurate depth from a rectified pair of stereo images (Han et al. (2015; Mayer et al. 2016a Kendall et al. 2017 Chang & Chen 2018). Kendall 2017) and Chang & Chen (018) design network architectures specifically for stereo matching. First, they apply 2D convolutional network extract learned features, then build a cost volume over the learned features. They then apply 3-D convolutions to the cost volume to perform feature matching and regularization. A similar idea has been extended estimate 3D structure from mul- tiple views (Kar et al. 2017 Yao et al. 2018). In particular, MVSNet (Yao et al., 2018) estimates depth from multiple images. However, these works require known camera poses as input, while our method estimates depth from video where the motion of the camera is unknown and estimated during inference.
Several works have used deep networks to predict camera pose. Kendall et al. (0115) focus on the problem of camera localization, while other work (Zhou et al. 201 Vijayanarasimhan al.] 2017 Wang et al. 2017) propose methods which estimate camera motion between pairs of frames in a video. Networks for motion estimation have typically relied on generie network components whereas we formulate motion estimation least-squares optimization problem. Whereas prior work has focused on estimating relative motion between pairs of frames, we can jointly update the pose of variable number of frames.
Motion:
Depth and Motion:
Geometric information has served as a self- supervuupyy signal for many recent
works Vijayanarasimhan al., [017]] Zhou et al.] 2017 Wann et al. 2018 Yin & Shi 2018 Yang ct al., 2018 Godard et al.] [017]] Mahjourian et 2018). In particular, Zhou et al. (2017) and
Vijyanarasimhan et al. (2017) trained single-image depth network and pose network while supervising on photometric consistency. However, while these works use geometric principles for training. they do not use multiple frames to predict depth at inference.
DeMoN (Ummenhofer et al., 2017) and DeepTAM (Zhou et al. 2018)) where among the first works combine motion estimation and multi-view reconstruction into a trainable pipeline. DeMoN (Um-) menhofer et al. 2017) operates on two frames and estimates depth and motion in separate network branches, while DeepTAM (Zhou et al.) 2018) can be used on variable number of frames. Like our work and other classical SLAM framesworks (Engel et al. 2014 Newcombe et al.) 2011). Deep- TAM separates depth and motion estimation, however we maintain end-to-end differentiablity be- tween our modules. A major innovation of Deep'TAM was to formulate camera motion estimation in the form of incremental updates. each iteration, DeM renders the keyframe from a synthetic viewpoint, and predicts the residual motion from the rendered viewpoint and the target frame.
Estimating depth and camera motion can be naturally modeled as a non-linear least squares problem, which has motivated several works to include an differentiable optimization layer within network architectures (Tang & Tan 2018 Wang et al. 2018: Clark et al. 2018 Bloesch et al.] 2018). We follow this line of work, and propose the Flow-SE3 module which introduces direct mapping from 2D correspondence to a 6-dof camera motion update. Our Flow-SE3 module is different from prior works such as DeMon (Ummenhofer al., (017)) and DeepTAM (Zhou et al.) (0188) which do not impose geometric constraints on camera motion and use generic layers. BA-Net (Tang & Tan 2018) and LS-Net (Clark al. 2018) include optimization layers, but instead optimize over photometric error (either pixel alignment (Clark et al. 2018) or feature alignment (Tang & Tan 2018)). Our Flow-SE3 module still imposes geometric constraints on camera motion like BA-Net (Tang & Tan 2018), but we show that in minimizing geometric reprojection error ( ddifference of pixel locations), we end up with a well-behaved optimization problem, well-suited for end-to-end training.
An important difference between our approach and BA-Net is that BA-Net performs one joint opti- mization problem by formulating Bundle- Adjustmettttt as a differentiable network layer, whereas we separate motion and depth estimation. With this separation, Our final reconstructed depth is the product of a cost volume, which can adapt the reconstruction as camera motion updates improve, while the output of BA-Net is restricted by the initial quality of the depth basis produced by a single-image network.
avoid the need for a depth basis.
3

---

Published as a conference paper at ICLR 2020
Camana Pogss
Matching Network Hourglassi
Backprojection
View Pooling
Hourglssa):
Dopth Predictions
Koybam
Vilumes
Cost
2x3x3 Conw
Figure 2: The Depth Module performs stereo matching over multiple frames to estimate depth. First each image is fed through network to extract dense feature map. The 2D features are backprojected into a of cost volumes. The cost volumes processed by set of 3D hourglass networks to perform feature matching. The final cost volume is processed by the differentiable arg-max operator to produce pixelwise depth estimate.
APPROACH
DeD predicts depth from calibrated video sequence. We take video input and output dense depth. We consider two subproblems: depth estimation and motion estimation. Both subprob- lems formulated as trainable neural network modules, which refer to the Depth Module and the Motion Module. Our depth module takes camera motion as input and outputs an updated depth prediction. Our motion module takes depth as input, and outputs motion correction the forward pass, we alternate between the depth and motion modules as we show in Figure [
In
Notation and Camera Geometry: As a preliminary, define some of the operations used within the depth and motion modules. We define to be the camera projection operator which maps 3D point X (X, 2, 1)" image coordinates x (u, v). Likewise, -1 defined be the backprojection operator, which maps pixel x and depth to 3D point. Using the pinhole camera model with intrinsics (fa: fy, Cz. Gy) we have
X (X) = (fx (fz + Ca, co + cg),
(x, = -C fz
V cy fy
1)
(1)
The camera pose is represented using rigid body transform G E SF(3). To find the image coor- dinates of point X in camera i, we chain the projection and transformation: where G; is the pose of camera i.
(u, v) = (G,X),
Now, given two cameras G; and Gj. we know the depth of point x² = (u', o²) camera i, we find its reprojected coordinates in camera j:
T (G, - (x, =T===fe (G 'x (x,
(2)
using the notation G.j = G, G for the relative pose between cameras i and
3.1 DEPTH MODULE
The depth module takes collection of frames, I (I1, I2.... IN along with their respective pose estimates, G [G1, G2.... .... GN }. and predicts dense depth map D for the keyframe (Figure 2). The depth module works by building a cost volume over learned features. Information aggregated multiple viewpoints by applying global pooling layer which pools across viewpoints.
The depth module can be viewed as the composition of building blocks: 2D feature extractor, cost volume backprojection, and 3D stereo matching.
2D Feature Extraction: The Depth Module begins by extracting learned features from the input images. The 2D encoder consists of stacked hourglass modules (Newell et al.) 2016) which maps

---

Published as a conference paper at ICLR 2020
each image dense feature map I, + Fj. More information regarding network architectures is provided in the appendix.
Take 1 be the keyframe, cost volume is constructed for each of the remaining N-1 frames. The cost volume for frame j. C², is constructed by backprojecting 2D features into the coordinate system defined by the keyframe image. To build the cost volume, enumerate over a range of depths ž1, 22, ........... 2p which is chosen to span the ranges observed in the dataset (0.2m- 10m for indoor scenes). For every depth =kx use Equation to find the reprojected coordinates on frame j. and then use differentiable bilinear sampling of the feature map Fj.
Cost Volume Backprojection:
More formally, given a pixel x = (u, E N² frame 1r and depth zg:
Ouvk = G - (x, (x, =x))) Hxx
(3)
where F(-) is the differentiable bilinear sampling operator (Jaderberg et al. 2015). Since the bilinear sampling is differentiable. C is differentiable w.r.t all inputs, including the camera pose.
Applying this operation to each frame, gives us a set of N-1 cost volumes each with dimension HxWxDxC. a final step, we concatenate each cost volume with the keyframe image features increasing the dimension to Hx W xD.2C. By concatenating features, we give the network the necessary information to perform feature matching between the keyframe/image pairs without dec- imating the feature dimension.
The set of N-1 cost volumes are first processed by a series of 3D con- volutional layers to perform stereo matching. We then perform view pooling by averaging over the N-1 volumes to aggregate information across frames. View pooling leaves us with a single volume of dimension Hx W x The aggregated volume is then processed by series of 3D hourglass modules, each outputs an intermediate depth.
3D Matching Network:
Each 3D hourglass module predicts an intermediate depth estimate. We produce intermediate depth representation by first applying a 1xlx1 convolution to a produce Hx XD volume. We then apply the softmax operator over the depth dimension, that for each pixel, we get probability distribution over depths. We map the probability volume into a single depth estimate using the differentiable argmax function (Kendall al. 2017) which computes the expected depth.
3.2 MOTION MODULE
The objective of the motion module is to update the camera motion estimates given depth as input. Given the input poses, G {G1, G2...., Gn }. the motion module outputs set of local perturba- tions & (51, (2.,,...... EN) ), & E se(3) used to update the poses. The updates are found by setting up a least squares optimization problem which is solved using a differentiable in-network optimization layer.
Initialization: We use a generic network architecture to predict the initial pose estimates similiar to prior work Zhou et al. (207)). We choose one frame to be the keyframe. The poses are initialized by setting the keyframe pose to be the identity matrix, and then predicting the relative motion between the keyframe and each of the other frames in the video.
Feature Extraction: Our motion module operates over learned features. The feature extractor maps every frame to dense feature map, 1, F,. The weights of the feature extractor are shared across all frames. Network architecture details are provided in the appendix.
I,), with respective poses (G,, Gy) and feature maps (F,, Ff)) Given depth Z; we can use Equation 2 we can warp F, onto camera to generate the warped feature map F,. If the relative pose G. G, G- is correct, then the feature maps F; and F, should align. However, if the relative pose is noisy, then there will be misalignment between the feature images which should be corrected by the pose update.
Error Term: Take two frames. (I,, I,),
We concatenate F; and Fy. and send the concatenated feature map through an hourglass network to predict the dense residual flow between the feature maps, which we denote R. and corresponding confidence map W. Using the residual flow, we define the following error term:
(Es, of (E,, 5) rs ( G Gj)(e G.) -1x (G.x
X (x(, =k)
(4)
5

---

Published as a conference paper at ICLR 2020
Feature Extracion
Residual Flww Prediction
(5 83)
att-argmin
G1.
G Sx)
G
G
G
>G
Gx
Figure 3: The Motion Module updates the input pose estimates by solving a least squares optimiza- tion problem. The motion module predicts the residual flow between pairs of frames, and uses the residual terms to define the optimization objective. Pose increments & are found by performing a single differentiable Gauss-Newton optimization step.
where r3 the residual flow at pixel X; predicted by the network, and 2 is the predicted depth. The weighting map W is mapped (0, 1) using the sigmoid activation, and is used determine how the individual error terms are weighted in the final objective.
The previous section showed how two frames (i, j) can be used define a collection of error terms cg (E,,E) E₁) for each pixel X₃ in image I,. The final optimization objective is a weighted combination of error terms:
Optimization Objective:
E(E) = Σ Σ (Er, diag(ws) of Eg),Es), (5,))CC
diag(wy) =
(5)
This leaves us with the question of which frames pairs (i,j) E C to use when defining the optimiza- tion objective. In this paper, we consider two different approaches which we refer Global pose optimization and Keyframe pose optimization.
Our global pose optimization uses all pairs of frames C= (i.j), i + to define the objective function (Equation 5 and the pose increment & is solved for jointly over all poses. Therefore, given N frames, dense pose optimization uses NxN-1 frame pairs. Since every pair of frames is compared, this means that the global pose optimization requires the predicted depth maps for all frames as input. Although each pair (i.j) only gives us information about the relative pose Gij. considering all pairs allows to converge
Global Pose Optimization:
globally consistent pose graph.
Our keyframe pose optimization selects a given frame to be the keyframe (i.e select Iı as the keyframe), and only computes the error terms between the keyframe and each of the other frames: C = (1,j) for = N.
Keyframe Pose Optimization:
Fixing the pose of the keyframe, that each error e. (O, Ey) term is only function of single pose increment č,. Therefore, we solve for each of the - pose increments independently. Additionally, since = for all pairs (i,j) E C, we only need the depth of the keyframe input when using keyframe pose optimization.
remove &1 from the optimization objective. This means
LS- Os-Opiimization Layer: increments by applying Gauss-Newton update. We backpropogate through the Gauss-Newton update so that the weights of the motion module (both feature extractor and flow network) can be trained on the final objective function. the appendix, we provide additional information for how the update is derived and the expression for the Jacobian of Equation
Using the optimization objective in Equation 5
we solve for the pose
3.3 FULL SYSTEM
During inference, we alternate the depth and motion modules for a selected number of iterations. The motion module uses depth to predict camera pose. As the depth estimates converge, the camera poses become more accurate. Likewise, as camera poses converge, the depth module can estimate more accurate depth.

---

Published as a conference paper at ICLR 2020
Image ground-try.c trhhh
image
ground-trush
NYUv2
SUNED ScanNet*
Figure 4: Visualization of predicted depth maps on NYU, ScanNet, and SUN3D. On ScanNet and SUN3D (marked with *) we show the results of the model trained only on NYU data.
Initialization: We try two different strategies for initialization in our experiments: (1) self initial- ization initializes VeD with a constant depth map and (2) single image initialization uses output of single-image depth network for initialization. Both methods give good performance.
3.4 SUPERVISION
Depth Supervision: We supervise on the LI distance between the ground truth and predicted depth. We additionally apply small I.I smoothness penalty to the predicted depth map. Given predicted depth Z and ground truth depth Z", the depth loss is defined as:
Ldepth (2)
Σ [x(x)) Zd (xxx)]
Σ (0,
(6)
Motion Supervision: We supervise pose using the geometric reprojection error. Given predicted pose G and ground truth pose G", the pose loss is defined
Emotion
(G) = Σ | (x, Z(x))) xTG"T= Z(x) )))lla
X.
where || lls is the robust Huber loss; we set ô =
Total Loss: The total loss is taken as weighted combination of the depth and motion loss terms: Edepth Xmmotion where we set 1.0 in our experiments.
EXPERIMENTS
test V2D across a wide range of benchmarks to provide a thorough comparison other methods. While the primary focus of these experiments is to compare to other works which estimate depth from multiple frames, often single-view networks still outperform multiview depth estimation. To put results in proper context, comparisons. Since not possible to recover the absolute scale of the scene through SfM, we report all results (both ours and all other approaches) using scale matched depth (Tang & Tan 2018).
include both multiview and state-of- -ot---ofthhennn single- image
Our primary experiments are on NYU, ScanNet, SUN3D, and KITTI, and we report strong results across all datasets. We show visualization of our predicted depth maps in Figure 4 The figure shows that V2D can recover accurate and sharp depth even when applied to unseen datasets. One aspect of particular interest cross-dataset generalizability. Our results show that DeepV2D generalizes very well-we we achieve the highest accuracy on ScanNet and SUN3D even without train- ing on either dataset.
4.1 DEPTH EXPERIMENTS
We evaluate depth NYU Silberman et al. 2012), ScanNet (Dai al. 2017). SUN3D (Xiao ct al. 2013), and KITTI (Geiger et al.] (013). On all datasets, Deep is given a video clip with unknown camera poses and alternates depth and pose updates and is evaluated after iterations.
7

---

Published as a conference paper at ICLR 2020
NYU depth (Silberman et al., 2012) is dataset composed of videos taken in indoor settings including offices, bedrooms, and libraries. We experiment on NYU using the standard train/test split (Eigen et al. 2014) and report results in Table I using scaled depth (Zhou et al. (017: Tang & Tan 2018). We evaluate two different initialization methods of our approach. Self-ini uses a constant depth map for initialization, while fern-init uses the output of FCRN (Laina et al. 2016) single-
NYU:
view network for initialization. Using a single-image depth network for initialization gives slight improvement performance.
NYCx2
á < 1.25
6 1.25
6 1.25
Abs Rel 1
Sc Inv
RMSE 1
log10
multi- single
FCRN (Laina et al. 2016
0.853
0.965
0.991
0.121
0.151
0,592
0.052
DORN IFu al 2018
0.875
0,966
0.989
0.109
0.464
0.047
Alhashim Wonka 2018
0.895
0,980
0.996
0.103
0.390
0.043
COLMAP
0.619
0.760
0.829
0.312
LS12
1,381
0.153
DIUSMC
0.487
0.697
0.814
0.447
0.456
1.793
0.169
MVSNet+ OpenMVG
0.766
0,913
0.965
0.181
0.212
0.917
0.072
DeMoN
0.776
0,933
0.160
0.196
0.775
DeMoN
0.805
0.951
0.985
0.144
0.179
0.717
0.061
Ours (selF-imit) - KKyyfamm
0,940
0.985
0.995
0.072
0,105
0.459
0.031
Ours (fern-init)- Keyframe
0.955
0,990
0.996
0.062
0.095
0.405
0.027
Ours (self-init) Global
0.942
0.986
0.995
0.070
0.104
0.454
0.030
Ours (Eem--init) Global
0.956
0.989
0.996
0.061
0.094
0.403
0.026
Table I: Results the NYU dataset. Our approach outperforms existing single-view and multi-
view depth estimation methods. Ours (self-init) uses a constant depth map for initialization while ours(fern-ini)) uses single-image depth network for initialization.
We compare to state-of-the-ar single-image depth networks DORN (uu et al. 2018 and DenseDepth (Alhashim & Wonka 2018) which are built on top of pretrained ResNet (DORN) or DenseNet-201 (DenseDepth). The results show that we can do much better than single-view depth by using multiple views. We also include classical multiview approaches such as COLMAP (Schonberger & Frahm 2016a) and DIUSMC (Ha et al., (016) which estimate poses with bundle adjustment, followed by dense stereo matching. While COLMAP uses SIFT features, DfUSMC built on local-feature tracking and is designed for small baseline videos.
Table also includes results using multi-view deep learning approaches. MVSNet (Yao et al. 2018) is trained to estimate depth from multiple viewpoints. Unlike our approach which estimates camera pose during inference, SVeN requires ground truth poses as input. We train MVSNet on NYU and use poses estimated from OpenMVG (Moulon et al.) during inference. Finally, we also evaluate DeMoN (Ummenhofer et al., 2017) on NYU. DeMoN is not originally trained on NYU, but instead trained a combination of 5 other datasets. We also try version of DeMoN which we retrain on NYU using the code provided by the authors (denoted t).
In Appendix C, we include additional results on NYU where test different versions of our model, along with parameter counts, timing information, peak memory usage, and depth accuracy. A shal- lower version of DeepV2D (replacing the stacked hourglass networks with single hourglass net- work) and lower resolution inference still outperform existing work on NYU. However, using a 3D network for stereo matching turns out to be very important for depth accuracy. When the 3D stereo network is replaced with a correlation layer (Dosovitskiy et al. 2015) and 2d encoder-decoder, depth accuracy worse increasing Abs-Rel Rel from 0.062 to 0.135.
Figure 3 shows the impact of the number of iterations and views on the scale-invariant (sc-inv) validation set accuracy. Figure 5 (left) shows that Deep requires very few iterations to con- verge, suggesting that block coordinate descent is effective for estimate depth from small video clips. In Figure 5 (right) we test accuracy as a function of the number of input frames used. Al- though V2D is trained using a fixed number (4) frames as input, accuracy continues to improve a more frames are added.
ScanNet:
ScanNet is large indoor dataset consisting of 1513 RGB-D videos in distinct scenes. We use the train/test split proposed by 'Tang & Tan (018) and evaluate depth and pose accuracy in Table 2 While primary focus is depth, Deep V2D accurately predicts camera motion.
We use ScanNet to test cross-dataset generalization and report results from two versions of our approach: ours (nyu) is method trained only on nyu, ours (scannet) is method trained on ScanNet. As expected, when we train on the ScanNet training set we do better than we train only on NYU. But the performance of our NYU model is still good and outperforms BA-Net on
8

---

Published as a conference paper at ICLR 2020
0.20
ours (fcrn-init)
0.14
(self-init) fcrn
0.12
Sc-inv
0.15
Sc-inv
0.10
0.10
0.08
0.06
0.05
1
2
4
6
2 3
5
7
iteration
frames
Figure 5: Impact of the number of iterations (left) and frames (right) sc-inv validation accuracy. (left) shows that V2D quickly converges within a small number of iterations. (right) we see that accuracy consistently improves more views are added. DeepV2D can be applied to variable numbers of views for a variable number of iterations without retraining.
ScanNet
Abs Rel
Sq Rel |
RMSE
RMSE log
se inv
rot(dgg)
tr. (deg)
tr. (cm)
DeMoN
0,231
0.520
0.761
0.289
0.284
3.791
31.626
15.50
BA-Net (orig.)
0.161
0.092
0.346
0.214
0.184
L018
20.577
3.390
BA-Net (5-view)
0.091
0.058
0,223
0.147
0.137
1.009
14.626
2.365
DSO (Engel et al. 2018
0.925
19.728
2.174
DSO (cm--mil)
Ours (nyu)
0.080
0.018
0.223
0,109
0.105
0.714
12.205
1.514
Ours (scannet)
0.057
0.010
0.168
0,080
0.077
0.628
10.800
1.373
Table ScanNet experiments evaluating depth and pose accuracy and cross-dataset generalization. Our approach trained on NYU (ours nyu) outperforms BA-Net despite BA-Net being trained on ScanNet data; training on ScanNet (ours scannet) gives even better performance.
all metrics. The design of our approach is motivated by generalizability. Our network only needs learn feature matching and correspondence; this experiment indicates that by learning these low level tasks, we can generalize well to new data.
Pose accuracy from DSO ESSeel et al.) (2018) is also included in Table 2 We test DSO using both the default initialization and single-image depth initialization using the output of FCRN (Laina et al. 2016). DSO fails initialize or loses tracking on some of the sequences so we only evaluate on sequences where DSO is successful. DSO fails on 335 of the 2000 test sequences while DSO (fern-init) fails on only 271.
SUN3D: SUN3D (Xiao et al. 2013) is another indoor scenes dataset which we use for comparison with De. DeepTAM only evaluates their depth module in isolation using the poses provided by dataset, while our approach is designed to estimate poses during inference. We provide results from our SUN3D experiments Table 3
SUN3D
Training Data
LI-Inv
LI-Rel J
Se-Inv |
SGM
0.197
0.412
0.340
DTAM
0.210
0.423
0.374
DeMoN
BIL-RGBD-MVS-SUN MVS-SUNCG+SLN3D
0,146
DecpTAM
0.101
0.128
NYU NYU ScanNet
0.056
0.106
0.134
Ours
0.041
0.077
0.104
Table Results on SUN3D dataset and comparison DeePTAM.. DeepTAA only evaluates depth in isolation and uses the poses from the dataset during inference, while our approach jointly estimates camera poses during inference. We outperform DeepTAM and DeMoN on SUN3D even when we do not use SUN3D data for training.
We cannot train using the same data as DeepTAM since DeepTAM is trained using combination of SUN3D, SUNCG, and MVS, and, at this time, neither MVS nor SUNCG are publicly available. Instead we train alternate data and test SUN3D. We test two different versions of our model; one where we train only on NYU, and another where we train on a combination of NYU and ScanNet data. Our NYU model performs similiar to DeepTAM; When combine with ScanNet data, outperform DeepTAM even though Dec is trained on SUN3D and is evaluated with ground truth pose input.

---

Published as a conference paper at ICLR 2020
The KITTI dataset (Geiger et al. 2013) is captured from moving vehicle and has been widely used to evaluate depth estimation and odometry. We follow the Eigen train/test split (Eigen et al. 2014), and report results in Table 4 We evaluate using the official ground truth depth maps.
KITTI:
We compare to the state-of-the-ant single-view methods and also multiview approaches such as BA-
Net (Tang & Tan 2018), and outperform previous methods the KITTI dataset across all metrics.
KITTI
Multi
1.25 1
8 < 1,25"
6 < 1.25
Abs Rel
Sy Rel 1.
Sq Rel
RMSE
RMSE log J.
DORN
0.988
DIUSMC
0.617
0.796
0.874
0.346
5.984
8,879
0.454
BA-Net
0.025
Ours
0.977
0.993
0.997
0.037
0.174
0.013
2.005
0.074
Table 4: Results on the KITTI dataset. We compare to state-of-the-art single-image depth network
DORN (Fu et al. 2018) and multiview BA-Net (Tang & Tan 2018). BA-Net reports results using different form of the Sq-Rel metric which we denote t.
Overall, the depth experiments demonstrates that imposing geometric constraints on the model ar- chitecture leads to higher accuracy and better cross-datases generalization. By providing differ- entiable mapping from optical flow to camera motion, the motion network only needs to learn to estimate interframe correspondence. Likewise, the 3D cost volume means the the depth network only needs to learn to perform stereo matching. These tasks are easy for the network to learn, which leads strong results all datasets, and can generalize to new datasets.
4.2 TRACKING EXPERIMENTS
DeepV2D can be turned into basic SLAM system. Using NYU and ScanNet for training, we test tracking performance on the TUM-RGBD tracking benchmark (Table using sensor depth as input. We achieve a lower translational rmse [m/s] than DeepTAM on most of the sequences. DeepTAM uses optical flow supervision to improve performance, but since network directly maps optical flow to camera motion, we do not need supervision on optical flow.
We use our global pose optimization in our tracking experiments. We maintain fixed window of 8 frames during tracking. At each timestep, the pose of the first 3 frames in the window are fixed and the remaining 5 are updated using the motion module. After the update, the start of the tracking window is incremented by I frame. We believe our ability to jointly update the pose of multiple frames is key reason for our strong performance on the RGB-D benchmark.
360
desk
desk2
plant
ruom
rpy
xyz
mean
DVO Kerl et al.
2013
0.125
0.037
0.020
0.062
0.042
0.082
0.051
0.060
Dee Zhou et al
2018
0.054
0.027
0.017
0.057
0.039
0.065
0.019
0.040
Deep'TAM (wo flow) ) Zhou ct al. 2018
0.069
0.042
0.025
0.063
0.051
0.070
0.030
0.050
Ours
0.046
0.034
0.017
0.052
0.032
0.037
0.014
0.033
Table 5: Tracking results in the RGB-D benchmark (translational rmse [m/s]).
5 CONCLUSION
We propose DeepV2D, deep learning architecture which is built by composing classical geometric algorithms into fully differentiable pipeline. DeepV2D is flexible and performs well across variety of tasks and datasets.
Acknowledgements We would like to thank Zhaoheng Zheng for helping with baseline experiments. This work was partially funded by the Toyota Research Institute, the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research (OSR) under Award No. OSR- 2015-CRG4-2699, and the National Science Foundation under Grant No. 1617767.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In OSDI, volume 16, pp. 265-283, 2016.
10

---

Published as a conference paper at ICLR 2020
Sameer Agarwal, Keir Mierle, et al. Ceres solver. 2012.
Ibraheem Alhashim and Peter Wonka. High quality monocular depth estimation via transfer learn- ing. arXiv preprint arXiv:1812.11944 2018.
Michael Bloesch, Jan Czarnowski, Ronald Clark, Stefan Leutenegger, and Andrew J Davison. crnsslam-lcarning compact, optimisable representation for dense visual slam. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2560-2568, 2018.
Jia-Ren Chang and Yong-Sheng Chen. Pyramid stereo matching network. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5410-5418, 2018.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear
memory cost. arXiv preprint arXiv: 1604.06144, 2016.
Ronald Clark, Michael Bloesch,
Czarnowski, Stefan Leutenegger, and Andrew J Davison. Learning solve nonlinear least squares for monocular stereo. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 284-299, 2018.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießnner. Scannet: Richly-annotater 3d reconstructions of indoor scenes. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition, pp. 5828-5839, 2017.
Alexey Dosovitskiy, Philipp Fischer, Eddy llg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, pp. 2758-2766, 2015.
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with common multi-scale convolutional architecture. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2650-2658, 2015.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using multi-scale deep network. In Advances neural information processing systems, pp. 2374, 2014.
Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In European Conference on Computer Vision, pp. 834-849. Springer, 2014.
Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct sparse odometry. IEEE transactions on paltern analysis and machine intelligence, (143):611-2255 2018.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2002-2011 1, 2018
Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE trans- actions on pattern analysis and machine intelligence, 3266):1362 2010.
Yasutaka Furukawa, Carlos Hernández, et al. Multi-viww stereo: A tutorial. Foundations and Trends& in Computer Graphics and Vision, 91-2):11-148 2015.
kitti dataset. The International Journal of Robotics Research, 32(11):1211-1331 2013. Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The
Riccardo Gherardi, Michela Farenzena, and Andrea Fusiello. Improving the efficiency hierar- chical
In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pp. 1594-1600. IEEE, 2010.
Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estima- tion with left-right consistency. In CVPR, volume 2, pp. 2017.
Hyowon Ha, Sunghoon Im, Jaesik Park, Hae-Gon Jeon, and In So Kweon. High-quality depth from uncalibrated small motion clip. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5413-5421, 2016.
11

---

Published as a conference paper at ICLR 2020
Xufeng Han, Thomas Leung, Yangqing Jia, Rahul Sukthankar, and Alexander C Berg. Matchnet: Unifying feature and metric learning for patch- based matching. Computer Vision and Pattern IEEE, 2015.
Recognition (CVPR), 2015 IEEE Conference on, pp.
Max Jaderberg. Karen Simonyan, Andrew Zisserman, et Spatial transformer networks. Ad- vances in neural information processing systems, pp. 2017-2025, 2015.
Abhishek Kar, Jitendra Malik, and Christian Häne. Learning a multi-view stereo machine. In Advances in Neural Information Processing Systems, pp. 364-375, 2017.
Alex Kendall and Roberto Cipolla. Geometric loss functions for camera pose regression with deep learning. Proc. CVPR, volume 3, pp. 8, 2017.
Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: A convolutional network for real- time 6-dof camera relocalization. In Computer Vision (ICCV), 2015 IEEE International Confer- ence pp. 2938-29929 2946. IEEE, 2015.
Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. End-to-end learning of geometry and context for deep stereo regression. In Proceedings of the IEEE International Conference on Computer Vision, pp. 66 75,75, 2017.
Christian Kerl, Jurgen Sturm, and Daniel Cremers. Dense visual slam for rgb-d cameras. In Intel- ligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on, pp. 2100-2106. IEEE, 2013.
Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 3D Vision (3DV), 2016 Fourth International Conference on, pp. 239- 248. IEEE, 2016.
Vincent Lepetit, Francese Moreno- Noguer, and Pascal Fua. Epnp: An accurate o (n) solution the pnp problem. International journal of computer vision, 81(2):155, 2009.
Shiqi Li, Chi Xu, and Ming Xic. A robust o (n) solution to the perspective-n-point problem. JEEE transactions on pattern analysis and machine intelligence, 34(7):1444-1455 2012.
H Christopher Longuet-Higgins A computer algorithm for reconstructing scene from two projec- tions. Nature, 293(5828) 1981.
David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
Reza Mahjourian, Martin Wicke, and Anelia Angelova. Unsupervised learning of depth and ego- motion from monocular video using 3d geometric constraints. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 5667-5675, 2018.
Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4040-4048, 2016a.
Nikolaus Mayer, Eddy Ilg. Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4040-4048, 2016b.
Roger Mohr, Long Quan, and Françoise Veillon. Relative 3d reconstruction using multiple uncali- brated images. The International Journal of Robotics Research, 14(6):619-322, 1995.
Pierre Moulon, Pascal Monasse, Renaud Marlet, and Others. Openmvg. an open multiple view geometry library.
Raul Mur-Artal and Juan Tardós. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Transactions on Robotics, 1255-1122222 2017.
12

---

Published as a conference paper at ICLR 2020
Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: versatile and accu- rate monocular slam system. IEEE Transactions on Robotics, 31(5):1147-1163, 2015.
Richard A Newcombe, Steven J Lovegrove, and Andrew Davison. Dtam: Dense tracking and mapping in real-time. In 2011 international conference computer vision, pp. 2320-2322 IEEE, 2011.
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estima- tion. In European Conference Computer Vision, pp. 483 499. Springer, 2016.
René Ranftl, Vibhav Vineet, Qifeng Chen, and Vladlen Koltun. Dense monocular depth estimation in complex dynamic scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2016.
Johannes L Schonberger and Jan-Michael Frahm. Structure-from-mootd motion revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4104 4113, 2016a.
Johannes L Schonberger and Jan-Michael Frahm. Structure-frommmottie revisited. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4104 4113, 2016b.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. Computer Vision-ECCV 2012. pp. 746-760, 2012.
Keith Snavely. Scene reconstruction and visualization from internet photo collections. 2009.
Noah Snavely. Scene reconstruction and visualization from internet photo collections: A survey. IPSJ Transactions Computer Vision and Applications, 3:44-66, 2011.
Deqing Sun, Xiaodong Yang, Ming-Yo Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8934-8943, 2018.
Chengzhou Tang and Ping Tan. Ba-net: Dense bundle adjustment network. arXiv preprint arXiv:1806.04880 04807, 2018.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26~ 31, 2012.
Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg. Alexey Doso- vitskiy, and Thomas Brox. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5038- 2017,
Sudheendra Vijayanarasimhan, Susanna Ricco, Cordelia Schmid, Rahul Sukthankar, and Kate- rina Fragkiadaki. Sfm-net: Learning of structure and motion from video. arXiv preprint arXiv: 1xXiv:7704.077 078044 2017.
Chaoyang Wang, José Miguel Buenaposada, Rui Zhu, and Simon Lucey. Learning depth from monocular videos using direct methods. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2022-2030 2030, 2018.
Sen Wang, Ronald Clark, Hongkai Wen, and Niki Trigoni. Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pp. 2043-2050 IEEE, 2017.
Changchang Wu et al. Visualsfm: A visual structure from motion system. 2011.
Jianxiong Xiao, Andrew Owens, and Antonio Torralba. Sun3d: A database of big spaces recon- structed using sfm and object labels. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1625-1632. 2013.
Nan Yang, Rui Wang. Jorg Stuckler, and Daniel Cremers. Deep virtual stereo odometry: Leveraging deep depth prediction for monocular direct sparse odometry. Proceedings of the European Conference Computer Vision (ECCV), pp. 817-833, 2018.
13

---

Published as a conference paper at ICLR 2020
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstruc- tured multi-view stereo. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 767-783, 2018.
Zhichao Yin and Jianping Shi. Geonet: Unsupervised learning of dense depth, optical flow and cam- era pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1983-1992 2018.
Huizhong Zhou, Benjamin Ummenhofer, and Thomas Brox. Deeptam: Deep tracking and mapping. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 222-838, 2018.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1851-1858, 2017.
A
APPENDIX
A.I LS-OPTIMIZATION LAYER:
In Equation 4 we defined the residual error to be:
1 (E,,C)) = rı - G.)
((,)
X (xic; 2k.
(8)
and the objective function as the weighted sum of error terms:
E(E) = Σ Σ (EI, diag(wa) of (Er,E))
diag( (wig(ww))=
(9)
wx
(N.))C
We apply a Gauss-Newtor update to Equation The Gauss-Newton update is computed by solving for the minimum the second order approximation the objective function:
Örp ŌE
= (J) W EN,),
Jp
6-
(10)
where r(f1, EN) is the stack of residuals and J is the Jacobian matrix. Each row J, is the Jacobian of the 1th error term w.r.t to each of the parameters. Each & is 6-dimensional, so optimizing over N poses means are updating 6.N variables.
Let rp cx (Ex, be the td residual, then
Det! (E): 0E,
Is (m((ss)))e&..)) - - (G,x
k==0,5==0
oe,
0 <(e Ge)
T( (G,)))))
(11)
de,
0g,
Del (Et, En) 8E,
0 (
T (i.
c (,,
S 0
0g,
Likewise, the Jacobian for Ej is
De (5ir 0e;
[r, - (1 (G)
- m G.y
-==0,,==0
Dg.
(12)
= n G, G
x
n Gije ((x()
05,
05.
0g.
using the adjoint to the increment the left of the transformation
14

---

Published as a conference paper at ICLR 2020
0
7(
t G
where Addjas
gg
(13)
Dek (5),E,)
-.=0,e,=0
(
c" (G, Adja.,
d5,
a
dw
where the Jacobian of the action of a SE(3) element 3D point is computed
de x
0
0
Z
Y
5=
0
1
0
Z
0
X
(14)
d
0
1
-Y
X
0
During training, we propagate through the Gauss-Newton update. The update is found by solving the linear system
- = H b J'J'Wr((I......
(15)
Since H is positive definite, we solve Equation 13 using Cholesky decomposition. In the backward pass, the gradients can be found by solving another linear system.
OC 0H
0L a
OC
OL
H
Y's,
H
(16)
a
B TRAINING DETAILS
V2D is implemented in Tensorflow (Abadi al. 2016). All components of the network trained from scratch without using any pretrained weights. We use gradient checkpointing (Chen) et al. 2016 to reduce memory usage and increase batch size.
When training on NYU and ScanNet, we train with 4 frame video clips. On KITTI, we use 5 frame video clips. The video clips created by first selecting keyframe. The other frames randomly sampled from the set of frames within a specified time window the keyframe. For example, on NYU, we create the training video by sampling from frames within second of the keyframe.
Training occurs the following two stages:
Stage I: We train the Motion Module using the Lmotion loss with RMSProp (Tieleman & Hinton
2012) and a learning rate 0.0001. For the input depth, we use the ground truth depth with missing values interpolated. We train Stage for 20k iterations NYU, 16k iterations KITTI, and 30k iterations on ScanNet.
Stage II: In stage II, we jointly train the motion and depth modules end-to-end on the combined loss with RMSProp. The initial learning rate is set to .001 and decayed to ,0002 after 100k training steps. During the second stage store depth predictions be used during the training epoch. We train Stage II for a total of 120k iterations with batch size of 2. In our ScanNet experiments, train for an additional 60k iterations.
Data Augmentation:
We perform data augmentation by adjusting brightness, gamma, and per- forming random scaling of the image channels. We also randomly perturb the input camera pose to the Motion Module by sampling small perturbations.
C TIMING AND MEMORY USAGE
In the below table provide timing and peak memory usage for different versions of our method. All results are obtained using 8 frame video sequences as input with the exception of the basline single-image network FCRN Laina al. (2016) which uses single frame input.
15

---

Published as a conference paper at ICLR 2020
Abs-Rel +
Parameters
Peak GPU Memory
Iteration Time
FCRN (Laina et al. 2016)
0.121
64M
0.1G
0.05s
Ours (1/2 res)
0.083
32M
0.7G
0.22s
Ours (1-HG)
0.071
16M
2.8G
0.61s
Ours (corT)
0.135
25M
1.8G
0.32s
Ours
0.062
32M
2.8G
0.69s
Table 6: Timing and memory details for different versions of our approach.
In ours(I-HG) we replace the feature extractor with a single 2D-hourglass network, and replace the stereo network with single 3D-hourglass network. The shallower network still performs well, but causes Abs-Rel to increase from 0.065 to 0.071, showing that stacking hourglass networks is beneficial for performance. In (1/2 res) test the performance of DeepVVD when images downsampled to 1/2 resolution for training and inference. Using lower resolution images decreases memory usage and inference time but slightly decreases accuracy.
We also test a version where we replace the 3d stereo network with a correlation layer and 2d encoder-decoder. In ours(corr), take the correlation between features the same depth range as we use to build the 3D cost volume, then concatenate the correlation response with features from the keyframe image, similar to DispNet (Mayer et al.) 2016b). The correlation version per- forms worse, increasing Abs-Rel from 0.065 to 0.135. This is consistent with prior work which has demonstrated that cost volumes give better performance than direct correlation (Kendall et al., 2017 Chang & Chen 2018).
D ADDITIONAL TRACKING INFORMATION
In Table we report tracking results for all sequences in the Freiburg dataset.
Sequence
RGB-D SLAM
DeepTAM
Ours
360
0.119
0.063
0.056
360(v)
0.125
0,054
0.046
desk
0.030
0.033
0.029
desk(v)
0.037
0.027
0.034
desk2
0.055
0.046
0.041
desk2(v)
0.020
0.017
0.017
floor
0.090
0.081
0.064
plant
0.036
0.027
0.019
plant(v)
0,062
0.057
0.052
room
0.048
0.040
0.047
room(v)
0.042
0,039
0.032
rpy
0.043
0.046
0.039
rpy(v)
0.082
0,065
0.037
teddy
0.067
0.059
0.043
xyz 0.051
0.019
0.025
xyz(v) 0.024
0.017
0.016
Average 0.058
0.043
0.037
Table 7: Per-Sequence tracking results the RGB-D benchmark evaluated using translational RMSE [m/s]. We outperform DeepTAM and DVO on 12 of the 16 sequences and achieve lower translational RMSE averaged over all sequences. While DeepTAM requires optical flow supervi- sion to achieve good performance, we do not require supervision on optical flow since the relation between camera motion and optical flow embedded into our network architecture.
16

---

Published as a conference paper at ICLR 2020
E CAMERA POSE ABLATIONS
The focus of this work on depth estimation, but we are interested in how different methods for estimating camera pose impact the final performance. In Table 8 we test different methods for estimating camera pose on NYU. In each experiment, we replace the motion module of our trained network with the given alternative, and test the final results. We also report results from MVSNet (trained on NYU) using each SfM implementation.
COLMAP (Schonberger Frahm 2016a) and OpenMVG (Moulon et al.) are publicly available SfM implementations. They do not return results on all input sequences, so we only evaluate se- quences were they converge without an error. PWCNet+Ceres takes the output of an optical flow network, PWCNet (Sun et al., (2018), and performs joint optimization of depth and pose using the Ceres solver (Agarwal et al. 2012). Finally, we evaluate MVSNet (Yao et al. 2018) when the pose predicted by Deep is given input. Note that not all SfM implementations converge all sequences (success rate is reported in parenthesis) and we only evaluate the method on the frames in which it converges.
Depth
Motion
Abs-Rel
81 1
52
0g 1
MVSNet
Identity
0.419
0.382
0.681
0,859
Dee
Identity
0.362
0.460
0.756
0.901
MVSNet
COLMAP(274/55A
0.244
0.724
0.857
-0.925
V2D
COLMAP
0.199
0.741
0.878
0.940
MVSNet
OVénMVG (4222MG(442/654) (422/654)
0.181
0.766
0..13
".995
V2D
OpenMVG
0.173
0.774
0.913
0.963
MVSNet
PWC+Cères (634/654)
0.279
0.651
0.845
0.925
Dec
PWC+Ceres
0.274
0.664
0,846
0.925
MVSNet
DeepV2D (654/654)
0.101
0.885
0.970
0.990
DeepV2D (ours)
0.062
0.955
0.990
0.996
Table 8: Impact of pose estimation method on depth accuracy. Replacing our motion module with SfM degrades performance for both MVSNet and our approach.
We also show results of our method when the motion module is replaced with other methods for estimation motion. In all cases, using SfM results in worse performance. We observe that classical SfM is not robust enough to consistently produce accurate poses, which leads to large errors on the test set. MVSNet performs better using the poses estimated by our network, but still underperforms full system, showing the importance of differentiable alternation between pose and stereo.
17

---

Published as a conference paper at ICLR 2020
F ADDITIONAL RESULTS
DORN
Figure 6: Visualizations of depth predictions on KITTI dataset.
Image
GT
FCRN
DeMoN
Ours
Figure 7: Additional results the NYU depth dataset Silberman et al. (2012) using 7-frame video clips. We show results compared with Laina et al. (2016) and Ummenhofer et al. (2017).
18

---

Published as a conference paper at ICLR 2020
G NETWORK ARCHITECTURES
serriic Did
cenvier (6) a0:de2
Flatory BR
Fastbry
creiid (139 srlie
canda
RarDery 9-0
Fasbery
candaa pong
Coeviel (4)
Cindaa SISt
ghts
Figure 8: Motion Module Architecture: The Encoder(left) extracts a dense 1/4 resolution feature map for each of the input images. The Residual Flow Network (right) takes pair of feature maps and estimates the residual flow and corresponding weights. This residual flow is estimated with an encoder-decoder network, with skip connections formed by concatenating feature maps. Numbers in parenthesis correspond to the number of output channels for each layer.
2D Encoder
MalPoo Dzt
HD.20 0-11
Great p31
ReaCanda
GenDco
PaaConn
NC 23:0
3D Matching Network
Cree Vilano
holupurrpe
Figure 9: Depth Module Architecture: The 2D encoder (top) is applied each image in the video sequence. The 2D Encoder consists of a series of residual convolutions and Hourglass Networks. The hourglass networks process the incoming features maps multiple scales. The hourglass net- work is defined recursively (i.c. HG(n) contains lower resolution hourglass HG(n-1)). We use nested hourglass modules with feature dimension 64-128-11 192-19922299 226. The resulting feature maps from the 2D encoder are used to construct the cost volumes. The 3D matching network (bottom) takes collection cost volumes as input. After a 1xlxl convolutional layer and a 3x3x3 residual convo- lution, we perform view pooling, which aggregates information over all the frames in the video. The aggregated volume is then processed by a series of 3D hourglass networks, each of which outputs an intermediate depth estimate. The widths of the 3D hourglass is 32-80- 32-80-128-1 128-176.
19


Reviews:
[{"review_id": "SJlDS4yrcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed to estimate depth from a video sequence. In the model, the pipeline iteratively estimates motion and depth by separate modules, which can be trained in an end-to-end fashion. The numerical results show better scores than the other SOTA methods.  Overall, I think this is a useful work and may be considered for publishing.\n\nThe following are the detailed comments:\n1. The introduction of the related works is well written.\n2. The empirical comparison is quite thorough and demonstrates the proposed method using sequence and multi-frame is very useful.\n3. The method has good generalization.\n\n4. Besides the empirical results, I found the paper reads quite mechanical and provides very limited understanding if any. It would be much better if the authors can make further effort to understand where exactly the performance comes from. With a more complete story, the paper would have a better potential to last longer.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgjyQ7ccr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework for training machine learning models that simultaneously estimates depth of objects and poses of a single camera in a sequence of images from a single camera, in others words, a video.\nIn a video, to estimate depth of objects, which is a main objective of this paper, we need to know the relative position and rotation information of a single camera in a sequence of images in a video (motional information). However, to estimate the relative position and rotation information of the camera, we need to know a ground truth depth information of each objects in each image.\nThe main idea works just like EM or alternating optimization. The depth module estimates depth of objects in a sequence of images assuming the relative position and rotation information of a single camera is given. The motion module estimates motional information of a camera assuming the depth of each object is given.\nThe authors formulate the aforementioned two modules as neural networks, so that they can be trained end-to-end, and proposes various way of initializing the two modules.\n\nWhile the idea is simple, I think the paper is well-written and the experiments show a superior performance over existing approaches.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJe8GmY2FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a neural network architecture for joint depth and camera motion estimation on video sequences. The authors propose an architecture that incorporates classic principles from SfM, namely depth computation based on cost volumes and motion estimation based on the reprojection error of features. Extensive experiments on a variety of datasets are shown and support that this approach provides strong results.\n\n+ Principled approach that marries the best aspects of deep learning with classic principles from multi-view geometry.\n\n+ Well-written paper\n\n+ Generalizes well\n\n+ Significantly outperforms the state of the art\n\n+ Clearly shows that more views help\n\n+ Seems to be robust to initialization\n\n\nQuestion: What happens if more Gauss-Newton steps are made? Could one trade computation for quality here?\n\n\nMinor issues:\n\n- Equation (1), right: f_z should probably be f_x\n- Paragraph between Eq. (1) and (2): There seems to be something wrong with the typesetting of x^i=... (e.g. the equals sign)\n- Same for the paragraph before Eq. (1) and x=(u,v)\n\n\nSummary: This paper presents a well-engineered and non-trivial system that leverages principles from different fields in a very reasonable way. The results look great both qualitatively and quantitatively. The experiments are extensive and show a clear improvement over the state-of-the-art.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1x0lB_6qH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper pushes forward the research of deep learning based video 3D reconstruction by decomposing the problem in two-stages:\n1. Depth estimation from multi-view stereo\n2. Camera pose estimation from optical flow estimation and PnP SE3 pose,\nwhich turns out to achieve state-of-the-art performance on public datasets. \n\nIn general, the iterative procedure of this paper is similar to DeepTAM (Zhou et al., 2018), the major difference is that the camera pose is estimated by PnP with estimated Flow but not directly predicted from sub-network structures, which contributes to the higher tracking accuracy and further improves the accuracy of multi-view stereo depth estimation. \n\nA multi-view camera pose estimation (the global pose optimization in the paper) is also proposed to utilize the relative relation between all video frame pairs, which is difficult for fully CNN based pipeline and maximizes the use of the explicit PnP optimization. \n\nAfter that, a residual flow field is predicted using CNNs and camera poses is further refined by minimizing the geometric re-projection error.\n\nSo this paper focuses more on the camera pose estimation than the depth, which is a good start point to achieve better multi-view capabilities in a CNN framework.\n\n\nHowever, I still have several concerns for this paper:\n\n1. The full system is highly engineered and complicated. For example, the feature map is extracted from two hour-glass networks, which seems over-complicated for feature extraction, and the multi-view stereo network using four 3D hour-glass networks, which consumes a large amount of memories. So I would like to see the authors demonstrate the performance from in simpler settings, e.g. the feature map can be a single encoder-decoder and the multi-view stereo is done by correlation and 2D convolution. So I would like to see the inference time, peak memory consumption and the model size. Ablation studies will also help the readers to understand whether performance gain is from the pose estimation or the network capacities, which is unclear in the current paper (Appendix.D). \n\n2. The comparison with state-of-the-art conventional system is missing. For example, is the camera pose estimation better than initializing the DSO (Engel et al., 2018) with a monocular depth estimation? In real applications, if the performance gain is insignificant, the conventional method will still be a better choice because the CNN based methods are computationally expensive on platforms without powerful GPUs. I will not downgrade the rating if the performance gain is insignificant, but it is necessary to see the comparison.\n\n\n3. Even the difference is ignorable for performance, I hope the author could use adjoint when deriving the derivatives in Eq.(11) of Appendix A.1. The author can refer to this tutorial http://ethaneade.com/lie.pdf or the text book https://www.eecis.udel.edu/~cer/arv/readings/old_mkss.pdf.\n\nPS: For the current version with 10 pages, I lean to a borderline score, but I select 6 because 5 is not an option. I will keep or raise the current score if my concerns are addressed during rebuttal. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a CNN architecture for joint depth and camera motion estimation from videos. The paper presents a differentiable formulation of the problem to allow its end-to-end learning, and the reviewers unanimously find the proposed approach reasonable and agree that this is a solid paper. Some of the reviewers find the method itself to be too mechanical, but they all agree that this is a well-engineered solution.
Decision: Accept (Poster)