Global-Local Network for Learning Depth with Very Sparse Supervision
Authors: Antonio Loquercio
Alexey Dosovitskiy
Davide Scaramuzza
Venue:  (2020)
Keywords: Depth Perception
Learning from Sparse Supervision
Learning from Interaction.

Abstract: Natural intelligent agents learn to perceive the three dimensional structure of the world without training on large datasets and are unlikely to have the precise equations of projective geometry hard-wired in the brain. Such skill would also be  valuable  to  artificial  systems  in  order  to  avoid  the  expensive  collection  of labeled datasets, as well as tedious tuning required by methods based on multi-view geometry. Inspired by natural agents, who interact with the environment via visual and haptic feedback, this paper explores a new approach to learning depth from images and very sparse depth measurements, just a few pixels per image. To learn from such extremely sparse supervision, we introduce an appropriate inductive bias by designing a specialized global-local network architecture. Experiments on several datasets show that the proposed model can learn monocular dense depth estimation when trained with very sparse ground truth, even a single pixel per image. Moreover, we find that the global parameters extracted by the network are predictive of the metric agent motion.

Under review as a conference paper at ICLR 2020
GLOBAL-LOCAL NETWORK FOR LEARNING DEPTH WITH VERY SPARSE SUPERVISION
Anonymous authors Paper under double-blind review
ABSTRACT
Natural intelligent agents learn perceive the three dimensional structure of the world without training on large datasets and are unlikely to have the precise equations of projective geometry hard-wired in the brain. Such skill would also be valuable to artificial systems in order to avoid the expensive collection of labeled datasets, as well as tedious tuning required by methods based on multi-view geometry. Inspired by natural agents, who interact with the environment via visual and haptic feedback, this paper explores new approach learning depth from images and very sparse depth measurements, just few pixels per image. To learn from such extremely sparse supervision, we introduce an appropriate inductive bias by designing a specialized global-local network architecture. Experiments on several datasets show that the proposed model can learn monocular dense depth estimation when trained with very sparse ground truth, even single pixel per image. Moreover, we find that the global parameters extracted by the network are predictive of the metric agent motion.
INTRODUCTION
Understanding of the three-dimensional (3D) structure of the world is crucial for the functioning of intelligent agents: for instance, it supports path planning and navigation, as well as motion planning and object manipulation. Animals, including humans, obtain such three- ttree-dimensional understanding naturally, without any specialized training. By observing the environment and interacting with they learn to estimate distances to objects using stereopsis and variety of monocular cues 2]. including motion parallax, perspective, defocus, familiar object sizes. How could artificial systems acquire such spatial awareness?
This question inspired long line of work on algorithmically extracting 3D structures from their two-dimensional (2D) projections [3]. Classically, multi-view geometry is used to reconstruct the 3D coordinates of points given their corresponding projections in multiple images. These geometric methods, carefully engineered over decades, demonstrate impressive results in variety of settings and applications [4] 5 One downside of this class of approaches is that they are very sensitive to hyper-parameters, e.g. camera calibration, which generally require tedious tuning. Additionally, they do not exploit more subtle monocular cues, such as perspective, defocus or familiar object size.
In order to utilize such cues, supervised learning approaches train deep neural networks to predict depth maps from annotated datasets [6]. Since they are not strictly dependent on projective geometry, they can achieve impressive results on sequences with unknown, and potentially varying, camera model and parameters, but require considerable amount of training data 77]. Collecting such datasets an expensive process, which is generally repeated every time the application environment changes. To combine the respective advantages of learning and geometry, unsupervised approaches to depth estimation S train deep neural networks through projective geometry, eliminating the need to collect large annotated training datasets. These approaches are remarkably successful for some scenes, for instance driving, but have not yet been clearly demonstrated in other environments, such as indoors. Additionally, being based geometry, they inherit high sensitivity the camera model and parameters and the need for extensive tuning.
The present work is motivated by the following question: How can 3D perception be learned by an embodied agent without the explicit use of projective geometry or large training datasets? To make the problem tractable, we make two assumptions. First, motivated by the extensive evidence from

---

Under review as a conference paper at ICLR 2020
psychology and neuroscience on the fundamental importance of motion perception [T0 we provide pre-computed optical flow as an input to the depth estimation system. Optical flow estimation can be learned either from synthetic data [14 [3] [6] or from real data in unsupervised fashion 17 I8). Second, we explore the idea that perception gets coupled to 3D properties of the world via interaction with the environment. Instead of realistically modeling haptic sensing. simulate an agent equipped with a low-cost range sensor, similar to a rangefinder or a 2D LiDAR. These provide very sparse depth ground truth, just a few pixels per image. Together with the monocular video, this is the only training signal that the agent receives.
12 131,
In order to learn from such sparse annotations, we design a lightweight global-local network architec- ture (see Fig. [ consisting of two modules global and local inspired by camera pose estimation and triangulation in standard geometric pipelines. The global module takes as input two images and the optical flow between them and outputs a compact latent vector of "global parameters", which encodes the observer's motion between the frames and the camera parameters. The local module, consisting of compact fully convolutional network, combines the "global parameters" with the optical flow field to produce the final depth estimate. Such a global-local decomposition provides our architecture with inductive bias that is appropriate for learning from very sparse annotations.
Since the method is inspired by learning via interaction, we evaluate it on diverse indoor scenes. We demonstrate that our network learns to estimate depth when provided, at training time, ground-truth for as little as a single pixel per image, i.e., 0.002% of the agent's field of view. We compare against generic deep networks, classic geometry methods, and unsupervised learning approaches. Standard convolutional networks show good results when trained with dense depth ground truth, but their performance degrades dramatically in the very sparse data regime. Both classic and unsupervised learning approaches are generally strong, but exhibit performance drops when calibration parameters are unspecified or dynamically changing. In the sparse data regime, the proposed approach outper- forms all baselines thanks to its ability to train with sparse labels and its robustness to variations in the camera parameters.
2 RELATED WORK
The problem of recovering the three-dimensional structure of scene from its two-dimensional projections has been long studied in computer vision [99 20 21 22]. Classic methods are based on multi-view projective geometry (3] The standard approach is first find correspondences between images and then use these together with geometric constraints to estimate the camera motion between the images (for instance, with the eight-point algorithm (1)) and the 3D coordinates of the points (e.g., via triangulation (20).. Numerous advanced variations of this basic pipeline have been proposed [23 24 5 improving or modifying various its elements. However, key characteristics of these classic methods are that they crucially rely on projective geometry, require laborious sand-engineering, and are not able to exploit non- con-motion--elatee depth cues.
To make optimal use of all depth cues. machine learning methods can either be integrated into the classic pipeline, or replace it altogether. The challenge for supervised learning methods is the collection of training data: obtaining ground truth camera poses and geometry for large realistic scenes can be extremely challenging. An alternative is to train on simulated data, but then generalization to diverse real-world scenes can become an issue. Therefore, while supervised learning methods have demonstrated impressive results [6] 7 225 26), it is desirable to develop algorithms that function in the absence of large annotated datasets.
Unsupervised (or self-supervised) learning provides an attractive alternative to the label-hungry supervised learning. The dominant approach is inspired by classic 3D reconstruction techniques and makes use of projective geometry and photometric consistency across frames. Existing works
use various depth representations for this task: voxel grids 27 point clouds 29 30 triangular
meshes [31 or depth maps 33 33 34). In this work we focus on the depth map representation. Among the methods for learning depth maps, some operate in the stereo setup (given a dataset of
images recorded by a stereo pair of cameras) R2 33 while others address the more challenging monocular setup, where the training data consists of monocular videos with arbitrary camera motions between the frames [8] [4). Reprojection- based approaches can often yield good results, but they crucially rely on geometric equations and precisely known camera parameters (one notable exception being the recent technical report in (35 which learns the camera parameters automatically). In
2

---

Under review as a conference paper at ICLR 2020
Depth estimate d
Optical flow W
Optical flo essimation
Loss
g
Global parameters
Sperse ground d
I,. In
Global module
Local
Figure 1: Global-local model architecture. An image pair and estimated flow field first fed through the global module estimates the "global parameters" vector g, representing the camera motion. From these global parameters, the local module generates three convolutional filter banks and applies them to the optical flow field. The output of the local module is then processed by a convolution to generate the final depth estimate.
contrast, we do not require knowing the camera parameters in advance and do not rely on projective geometry. Also related to work depth completion methods [36], which learn to predict full depth maps given sparse ground-truth annotations. However, while they assume to observe ground-truth also test time, we predict depth maps from two images only. Using sparse annotations only for training has been applied mainly to semantic segmentation 377 where additional cues, e.g., object masks, facilitate learning.
ground-truth annotations only training time. At test time,
Several works, similar to ours, aim learn 3D representations without explicitly applying geometric 40). A scene, represented by one or several images, is encoded by deep network into a latent vector, from which, given target camera pose, a decoder network can generate new views of the scene. A downside of this technique is that the 3D representation is implicit and therefore cannot directly used downstream tasks such as navigation or motion planning. Moreover, at training time it requires knowing camera pose associated with each image. Our method, in contrast, does not require camera poses, and grounds predictions in the physical world via very sparse depth supervision. This allows us to learn explicit 3D representation in the form of depth maps.
equations 38 39
METHODOLOGY
Given two monocular RGB images I,, I2, with unknown camera parameters and relative pose, as well as the optical flow w between them, we aim to estimate dense depth map corresponding to the first image. We assume to have an artificial agent equipped with range sensor, which navigates through an indoor environment. By doing so, it collects a training dataset of image pairs, with depth ground truth d available only for extremely few pixels. Using this sparsely annotated dataset, we train a deep network (I, 12, w), with parameters 0, that predicts dense depth map d over the whole image plane. We now describe the network architecture detail.
3.1 MODEL ARCHITECTURE
An overview of the global- network architecture is provided in Figure [ The system operates on an image pair I, I and the optical flow (dense point correspondences) between them. this work, we estimate the flow field with an off-the-shelf optical flow estimation algorithm, which is neither trained nor tuned on our data.
The rest of the model is composed of two modules: a global module G that processes the whole image and outputs a compact vector of "global parameters" and a local module that applies a compact fully convolutional network, conditioned on the global parameters, the optical flow field. This design motivated both by classic 3D reconstruction methods and by machine learning considerations. Establishing analogy with classic pipelines, the global module corresponds to the relative camera pose estimation, while the local module corresponds to triangulation estimation of depth given the image correspondences and the camera motion. These connections described in more detail in

---

Under review as a conference paper at ICLR 2020
the supplement. From the learning point of view, we aim to train a generalizable network with few labels, and therefore need to avoid overfitting The local module is very compact and operates on transferable representation optical flow. The global network is bigger and takes raw images as input, but it communicates with the rest of the model only via the low-dimensional bottleneck of global parameters, which prevents potential overfitring.
The "global module" G is implemented by a convolutional encoder with global average pooling at the end. The network outputs a low-dimensional vector of "global parameters" g - G(I, I2, w). The idea is that the vector represents the motion of the observer, although no explicit supervision is provided enforce this behavior. While the optical flow alone is in principle sufficient for ego-motion estimation, we also feed the raw image pair to the network to supply it with additional cues.
The "local module" L takes as input the generated global parameters g, as well as the optical flow field. First, the global parameter vector is processed by a linear perceptron that outputs several convolutional filters banks, collectively denoted by 80 = (g).). Then, these filter banks are stacked into a small fully convolutional network Cep that is applied to the optical flow field. We append two channels of x- and y- image coordinates to the input w of Cip. as in CoordConv 41 The output of Cip is the final depth prediction = (w)
This design of the local module motivated by classic geometric methods: for estimating the depth of a point it is sufficient to know its displacement between the two images, its image plane coordinates, and the camera motion. In contrast to this standard formulation of triangulation. we intentionally make the receptive field of the network larger than 1 x 1 pixel, so that the network has the opportunity to correct for small inaccuracies or outliers in the optical flow input.
3.2 LOSS FUNCTION
d à This is a common representation in computer vision and roboties [42] (44, which allows naturally handle points and their uncertainty over a large range of depths. We use the L1 loss on the inverse depth, averaged over the subset P of the pixels that have associated ground truth inverse depth
Similarly to previous work 25
7. we define the loss on the inverse depth 2 =
Ldepth
P
To encourage the local smoothness of the predicted depth maps. we add an L1 regularization penalty on the gradient V2 = (8,2, 0zz) of the estimated inverse depth. Similarly classic structure from motion methods and unsupervised depth learning literature (33, we modulate this penalty according to the image gradients ÖI, allowing depth discontinuities to larger at points with large ØI;
Esmooth
10
- 8.11 + 10, -1,7711,
SER
with Ω representing the full image plane. The full training loss of our network is weighted sum of these two terms Listal = Ap + AaLomohhh-
3.3 MODEL DETAILS
In all our experiments the input images have resolution 256 192 pixels. Unless mentioned otherwise, ground truth depth is provided for single pixel of each image, but we also experiment with denser ground-truth signals. We use a pre-trained PWC-Net [16] for optical flow estimation.
We use the Leaky ReLU non- non-linearity in all networks. The global module is implemented by a 5-layer convolutional encoder with the number of channels growing from to 256, with stride 2 in the first 4 layers. The last 256-channel hidden layer is followed by convolution with 6 channels and global average pooling, resulting in the 6-dimensional predicted global parameter vector g. The local module consists of single linear perceptron which transforms g linearly to a 3.9K vector. Empirically, we did not find an advantage in utilizing a multi-layer non-linear perceptron in place of the linear operation. The resulting vector is split into three parts, which are reshaped into filter banks with kernel size 3 x and number of output channels 20, 10. 20, respectively. These filter banks, with Leaky ReL.Us in between, constitute the compact fully-convolutionnal depth estimation network
4

---

Under review as a conference paper at ICLR 2020
Abs-Inv
Abs-Rel
S-RMSE
Formula
Eall
d-di
La
Fig ( Elog)
D
Table 1: Metrics for quantitative evaluation of depth accuracy. i'is ground truth depth, d is the prediction. For convenience we denote Eing = log(d/d) log d log d. For all metrics, lower is better.
C.p- The 20-channel network output is then processed by single 3 x 3 convolutional layer to shrink the channels to 1. The design of this compact fully convolutional network has been inspired by the refinement layer used by previous works on supervised depth estimation [25]
We train the model with the Adam optimizer [43 with initial learning rate of 10-4 for a total of approximately 94K iterations with mini-batch size of 16. We apply data augmentation during training. Further details are provided in the supplement.
4 EXPERIMENTS
We design our evaluation procedure to address the following questions: (i) Is the proposed global- local architecture advantage compared to standard deep convolutional networks when learning with sparse depth ground truth? (ii) What is the robustness of our approach to dynamically changing camera parameters? (iii) Do the learned global parameters contain information about the camera motion between the frames? (iv) How does the proposed method compare to a state-of-the-ar unsupervised learned depth estimation method. Finally, we validate our design choices with ablation studies. Quantitative comparisons against classic structure from motion methods, as well as additional qualitative experiments, are provided in the supplementary material.
4.1 EXPERIMENTAL SETUP
We test the approach on three datasets collected in cluttered indoor environments, either real or simulated. Scenes11 1 25 is large synthetic dataset with randomly generated scenes composed of objects from ShapeNet 4 against diverse backgrounds composed of simple geometric shapes. SUN3D [45] is large collection of RGB-D indoor videos collected with Kinect sensor. RGB-D SLAM (46) is another RGB-D D dataset collected with Kinect in indoor spaces. For all datasets, we use the splits proposed by 225 We discard all image pairs with larger relative rotation than 14 per axis and larger relative translation than meter per axis. In order to simulate range observations by the agent, mask out all depth ground truth except for single pixel (unless mentioned otherwise).
As commonly done in two-view depth estimation methods (25 and in structure- frree-rroomooeeeeee motion meth- ods (47), resolve the inherent scale ambiguity by normalizing the depth values such that the norm of the translation vector between the two views is equal to 1. To quantitatively evaluate the generated depth maps, we adopt three standard error metrics summarized in Table [
4.2 LEARNING FROM VERY SPARSE GROUND TRUTH
We compare the proposed global-local architecture to strong generic deep models the encoder- decoder architecture of Eigen et al. (6]. the popular fully convolutional architecture DispNet [48), and the multi-scale encoder-decoder of Laina et al. (FCRN) 49 Note that for a fair comparison with our method, we provide all the baselines with both the image pair and the optical flow field. We additionally tune the models reach best performance on our task. The details of the tuning process are reported in Section 6.2.1 in the Appendix. We also compare to reduced-sized DispNet [48] (Small Ene- Enc-Dec), that has number of parameters similar to our model (including both the global and the local module). Its encoder consists of convolutions with (16, 32, 54, 128) filters, with sizes of (7, 5, 3, 3), and stride 2. Its decoder is composed of 4 up-convolutions with (128, 64, 32, 16) filters of size 3 and stride 1. Encoder and decoder layers connected through skip connections. Finally, compare against Struct2Depth [9] current state-of ofw-oo---thh-eeeee art system for unsupervised depth estimation.
As shown in Table 2 our approach outperforms all the baselines in the sparse supervision regime. Specifically, we outperform the architecture of Eigen et al. [6] on average by 53%. the architec- ture of Laina et al. [49 by 22.5%, and the fully convolutional DispNet by 20%. Indeed, due to
5

---

Under review as a conference paper at ICLR 2020
Image
GT Depth
Struct2Depth
DispNet
Ours
Scenest
SUN33
RGB-D
Figure 2: Qualitative comparison of the depth maps generated with the baselines and approach. Overall, Struc2Depth's predictions are generally poor homogenemus repetitive regions, while DispNet tends to over-smooth depth maps. In contrast, our method can predict fine details of the scene geometry.
Method
Scenesl1
SUN3D
RGB-D
Abs-Inv Abs-Rel
S-RMSE
Abs-Inv Abs-Rel
S-RMSE
Abs-Inv
Abs-Rel
S-RMSE
Eigen
6
0.045
0.57
0.77
0.072
0.82
0.38
0.046
0.54
0.37
DispNet
48
0.038
0.51
0.70
0.041
0.49
0.33
0.038
0.45
0.36
FCRN 49
0.041
0.52
0.74
0.047
0.44
0,042
0.45
0.35
Small Enc- Dec
0.046
0.66
0.83
0.064
0.73
0.45
0.049
0.58
0.46
Septh
99
0.058
0.95
0.81
0.037
0.44
0.27
0.037
0.44
0.48
Ours
0.031
0.43
0.61
0.035
0.37
0.25
0.033
0.37
0.33
Table 2: the sparse training regime, our method can efficiently learn to predict depth from single point supervision, outperforming significantly both standard architectures and unsupervised depth estimation systems. For all error metrics, lower is better.
wver-parametrization these baselines tend to overfit to the training points, failing to generalize to unobserved images and locations.
This is empirically demonstrated in Fig. 3 where we plot the depth loss on training points as function the number of iterations. Decreasing the size of the architecture address overfitting does not however solve the problem: the Small Enc-Dec, with number of parameters similar to network, achieves poor results, mainly due to its limited capacity.
Abs-Inv on Scenes
0.05
0.04
0.03
Our approach also achieves average 24% better error than the unsupervised depth estimation baseline [9] over all datasets and metries. Indeed, the considered datasets represent chal- lenge for geometry-based methods given the presence of large homogeneous regions, occlusions, and small baselines between views, which are typical factors encountered in indoor scenes. Noticeably, the performance of Struct2Depth on the SUN3D dataset is relatively good, boosted by the larger baseline be- tween views and the abundance of features.
0.02
Eigen
0.01
DispNer
Our:
20k
40k
60k
80k
Figure 3: For large networks, the loss training points (solid lines) signif- icantly higher than the validation loss (dashed lines). In contrast, our global- local architecture learns generalizible representations.
Fig. 4 Hanalyzes the performance of our and the DispNet archi- tectures (our strongest baseline) as function the number of observed ground-truth pixels per image. Unsurprisingly. both methods learn to predict accurate depth maps when dense annotations (D) are available. Decreasing the amount of supervision obviously leads performance drops. However, for method the enror increases on average by only 5% when going to sparser supervision, compared to 12% for the baseline, which leads large advantage over the baseline in the single-pixel supervision regime.

---

Under review as a conference paper at ICLR 2020
Abs-Inv
Abs-Rel
S-RMSE
Scenes
0.04
0.6
0.8
0.03
0.45
0.6
0.02
0.3
0.4
SUN3D
0.05
0.5
0.4
Ours
0.04
0.35
0.3
DispNet
0.03
0.2
0.2
0.04
0.5
0.4
RGB-D
0.03
0.35
0.3
0.02
0.2
0.2
D
16
16
Figure 4: Depth estimation errors with increasing number of training pixels image and dense supervision (D). When supervision gets sparser, method's performance degrades more gracefully than the baseline.
This shows that the global-local architecture provides an appropriate inductive bias for learning from extremely sparse depth ground-truth.
4.3 ROBUSTNESS TO DYNAMICALLY CHANGING CAMERA PARAMETERS
In practical applications camera internal parameters, such focal length, may change through time. Indeed, environmental changes like temperature, humidity and pressure could cause severe variations to their nominal value. Due to these variations, methods based on projective geometry, which are sensitive to the accuracy of calibration parameters, can experience large performance drops. Although the problem could be alleviated by automatic re-calibration, these changes would have to be detected in the first place and would require either collecting multiple views of an object [50 51 or additional sensing (22
We empirically study the robustness of our method and the baselines to dynamically changing camera intrinsics. In particular, randomly change, for each image pair, the horizontal and vertical focal lengths, as well as the center of projection, by to 20% of their nominal value. The unsupervised depth estimation baseline suffers the most from the uncertainty in the camera intrinsics. Indeed, its estimation error increases average by 26% with respect to the case in which camera parameters are correctly set. In contrast, as our approach does not explicitly rely on projective geometry, it does not exhibit such sensitivity to the camera parameters. Indeed, it experiences only a small decrease in performance, of approximately 5% with respect to the case where the intrinsics are fixed, since the learning problem becomes more challenging.
Method
Scenes1
SUN3D
RGB-D
Abs-Inv Abs-Rel
S-RMSE
Abs-Inv
Abs-Rel
S-RMSE
Abs-Inv
Abs-Rel S-RMSE
Struet2Depth 9
0.062
2.19
0.87
0.045
0.52
0.25
0.050
0.54
0.46
DispNet 48
0.039
0.57
0.70
0.041
0.46
0.27
0.046
0.56
0.38
Ours
0.034
0.51
0.61
0.034
0.43
0.24
0.036
0.40
0.33
Table 3: Depth estimation errors with camera intrinsics varying up to 20% of their nominal value between views. Based on projective geometry, unsupervised methods suffer the most from parameter uncertainty.
4.4 GLOBAL PARAMETERS AND THE CAMERA MOTION
According to the intuition behind our model, the global parameters should have information about the observer's ego-motion between the frames, and as such should be related to the actual metric camera motion. Here we study this relation empirically, by training camera pose predictor on the output of our global module, in supervised fashion. Note that this is done for analysis purposes only, after full model has been trained: training time the model has no access to the ground truth camera poses. Specifically, we add a small two-layer MLP with 256 hidden units on top of the global module that is either pre-trained with our method randomly initialized. We then either train the
7

---

Under review as a conference paper at ICLR 2020
Scenes
SUN3D
RGB-D
Method
rot
trans
rot
trans
rot
trans
Scratch-MIA
1.3
74.4
3.6
55.5
5.3
78.4
Pretrained-MBl
0.9
26.7
2.7
32.5
4.4
51.5
Scratch-Full
0.7
10.3
1.8
25.0
3.2
30.5
Pretrained-Full
0.7
9.2
1.7
22.4
3,2
28.7
KLT Matlab 25
0.9
14.6
5.9
32.3
12.8
49,6
8-point FlowFields
25
1.3
19.4
3.7
33.3
4.7
46.1
Table 4: Estimation of camera motion based on the global parameters estimated by our model. We initialize the global module either randomly (Scratch) or as trained with our approach (Pretrained). We then append small MLP and train supervised camera motion prediction by tuning either just the MLP (MLP) or the full network (Full). As reference, also report the performance of two classic approaches. We report rotation (rot) and translation (trans) errors degrees (since the translation vector normalized to 1, see
Lower better.
full network only the appended small MLP to predict the camera motion in supervised fashion (details of the training process are provided in the supplement).
Results in Table show that the global parameters indeed contain information about the camera pose. In both training setups pre-trained network substantially outperforms the random initialization: 17% to 64% error reduction across datasets and metrics when only tuning the MLP and up 11% error reduction when training the full system. Interestingly, our method also competitive against classic state-of-the-arr baselines for motion estimation 25
4.5 ABLATION STUDY
Our architecture based on several design choices that we now validate through an ablation study, In particular, we ablate the following components: (i) the use of optical flow an intermediate representation, (ii) the estimation of global variables to generate convolutional filters, (iii) the use of coordinate convolution in the fully convolutional network and (iv) the use of the image pair, in addition to optical flow, for the estimation of global parameters.
The results in Table 5 show that all components important and some have larger impact than others.
Abs-Inv
Abs-Rel
S-RMSE
The use of optical flow and coordinate convolution
are crucial since they both provide essential cues
Full Model
0.033
0.43
0.61
for depth estimation. However, a basic encoder-
Image Pair
0.033
0.45
0.62
decoder architecture (i.e. without global variables or
CoordConv
0.038
0.52
0.71
coordinate convolutions) underperforms even when provided with optical flow. Unsurprisingly. the least important factor providing the image pair the global module, since, when camera parameters are fixed, the optical flow a sufficient statistics of the observer's ego-motion.
Glob. Mod.
0.041
0.55
0.73
Flow
0.052
0.73
0.81
Table 5: Ablation study on the Scenes dataset.
CONCLUSION
Motivated by the way natural agents learn predict depth, propose an approach for training dense depth estimator from two unconstrained images given only very sparse supervision at training time and without the explicit use of geometry. We show that in cluttered indoor environments our global-local model outperforms state-of-the-art architectures for depth estimation by up to 20% in the sparse data regime.
We see several potential ways for further improving the performance of our approach. The method suffers from outliers in the flow field or when the baseline between views very small (see Appendix for detailed evaluation). The former problem could be solved by learning depth estimation not only from the sparse ground truth, but also from the photometric reprojection error. This raises the question of how make of photometric error without explicit geometric equations. The problem of small baselines could be alleviated by exploiting monocular cues. We see these directions as exciting avenues for future work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
[1] I. P. Howard and B. J. Rogers, Seeing in depth, Vol. 2: Depth perception. Press, 2002.
University of Toronto
[2] E. Goldstein and Brockmole, Sensation and perception. Cengage Learning, 2016.
[3] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge University Press, 2003, second Edition.
[4] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós, monocular SLAM system," " IEEE Trans. Robot., vol. 31, no. 5. pp. 1147-1163, 2015.
A versatile and accurate
[5] Schönberger and J.-M. Frahm, "Structure-from frotmmmoo-oo revisited," in JEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2016, pp. 4104-4113.
[6] D. Eigen, C. Puhrsch, and R. Fergus, "Depth map prediction from single image using multi-scale deep network," in Neural Inf. Process. Syst. (NeurIPS), 2014, pp. 2366-2374.
[7] K. Lasinger, R. Ranftl, K. Schindler, and V. Koltun, "Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dttaset transfer," arXiv: 1907iv1907.01344 2019.
[8] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, "Unsupervised learning of depth and ego- motion from video," in IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2017, pp. 6612-6619.
[9] V. Casser, S. Pirk, R. Mahjourian, and A. Angelova, "Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos," in Thirty-Thire AAAI Conference on Artificial Intelligence (AAAI-19), 2019.
[10] H. Wallach and D. O' C'Conelll, "The kinetic depth effect." Journal of experimental psychology vol. 45, no. 4, p. 205, 1953.
[11] G. Johansson, "Visual motion perception," Scientific American, vol. 232, 6, pp. 76-89, 1975.
[12] K. Nakayama and J. Loomis, "Optical velocity patterns, selocity-sensitive neurons, and space perception: a hypothesis," Perception, vol. 3, no. 1, pp. 63-80, 1974.
[13] J. Koenderink, "Optic flow," Vision research, vol. 26, no. 1, pp. 161-179, 1986.
[14] A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazırbaş, V. Golkov, P. van der Smagt, D. Cremers, and T. Brox, "FlowNet: Learning optical flow with convolutional networks." Int. Conf. Comput. Vis. (ICCV), 2015, pp. 2758-2766.
[15] E. 1lg. N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, "Flownet 2.0: Evolution of optical flow estimation with deep networks," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Jul 2017. [Online]. Available: http:///mb.nformatii. uni wtw.mmatiiiuii-ffiiuurrrgg.. www.bbbr..d////tiiiitooo////////
[16] D. Sun, X. Yang, M.-Y. Liu, and Kautz, "PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume," in IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2018.
[17] J. Yu, A. W. Harley, and K. G. Derpanis, "Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness," " in ECCV 2016 Workshops, 2016.
[18] Meister, J. Hur, and S. Roth, "Unflow: Unsupervised learning of optical flow with bidirec- tional census loss," in AAAI Conf. Artificial Intell., 2018.
[19] H. C. Higinnt "A computer algorithm for reconstructing scene from two projections," Nature, no. 293, pp. 133-135, 1981.
[20] R. I. Hartley and P. F. Sturm, "Triangulation." Computer Vision and Image Understanding, vol. 68, no. 2, pp. 146-157, 1997.
[21] R. I. Hartley, "In defense of the eight-point algorithm," IEEE Trans. Pattern Anal. Machine Intell., vol. 19, no. 6. pp. 580-593, 1997.
[22] B. Triggs, P. McLauchlan, R. Hartley, and A. Fitzgibbon, "Bundle adjustment - modern synthesis." in Vision Algorithms: Theory and Practice, LNCS. W. Triggs, A. Zisserman, and R. Szeliski, Eds., vol. 1883. Springer Verlag, 2000, pp. 298-372.
[23] H. Hirschmuller and D. Scharstein, "Evaluation of stereo matching costs on images with radiometric differences." IEEE Trans. Pattern Anal. Machine Intell., vol. 31, 9, 2009.

---

Under review as a conference paper at ICLR 2020
[24] R. A. Newcombe, S. J. Lovegrove, and A. Davison, "DTAM: Dense tracking and mapping in real-time," in Int. Conf. Comput. Vis. (ICCV), Nov. 2011, pp. 2320-2327.
[25] B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. llg. A. Dosovitskiy, and T. Brox, "Demon: Depth and motion network for learning monocular stereo," in IEEE Conf. Compul. Vis. Pattern Recog. (CVPR), 2017, pp. 5038-5047,
[26] H. Zhou, B. Ummenhofer, and T. Brox, "Deeptam: Deep tracking and mapping," in Eur. Conf. Comput. Vis. (ECCV), 2018.
[27] S. Tulsiani, T. Zhou, A. A. Efros, and J. Malik, "Multi-view supervision for single-view reconstruction via differentiable ray consisteney," in IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2017, pp. 209-217.
[28] S. Tulsiani, A. A. Efros, and J. Malik, "Multi-view consistency as supervisory signal for learning shape and pose prediction," in IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2018,
pp. 2897-2905.
[29] C. Lin, C. Kong, and S. Lucey, "Learning efficient point cloud generation for dense 3d object reconstruction," " in AAAI Conf. Artificial Intell., 2018, pp. 7114-7121.
[30] E. Insafutdinov and A. Dosovitskiy, "Unsupervised learning of shape and pose with differen- tiable point clouds," in Neural Inf. Process. Syst. (NeurIPS), 2018, pp. 2807-2817.
[31] H. Kato, Y. Ushiku, and T. Harada, "Neural 3d mesh renderer," in JEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2018, pp. 3907-3916.
[32] R. Garg, B. G. V. Kumar, G. Cameiro, and D.D. Reid, "Unsupervised CNN for single view depth estimation: Geometry to the rescue," in Eur. Conf. Comput. Vis. (ECCV), 2016, pp. 740-756.
[33] C. Godard, O. Mac Aodha, and G. J. Brostow, "Unsupervised monocular depth estimation with left-right consistency," in IEEE Conf. Comput. Vis. Pattern Recog, (CVPR), 2017, pp. 6602-6611.
[34] R. Mahjourian, M. Wicke, and A. Angelova, "Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints"" in IEEE Conf. Comput. Vis. Puttern Recog. (CVPR), 2018, pp. 5667-5675.
[35] A. Gordon, H. Li, R. Jonschkowski, and A. Angelova, "Depth from videos in the wild: Un- supervised monocular depth learning from unknown cameras," arXiv, vol. abss/904..00 1904.04998, 2019.
[36] A. Eldesokey, M. Felsherg, and F. S. Khan, "Confidence propagation through enns for guided sparse depth regression," IEFE transactions on paltern analysis and machine intelligence, 2019.
[37] A. Bearman, O. Russakovsky, V. Ferrari, and L. Fei-Fei, "What's 's the point: Semantic segmenta- tion with point supervision," in European conference on computer vision. Springer, 2016, pp.
549-565.
[38] M. Tatarchenko, A. Dosovitskiy, and T. Brox, "Multi-view 3d models from single images with a convolutional network,"" " in Eur. Conf. Comput. Vis. (ECCV), 2016. pp. 322-337.
[39] D. J. Rezende, S. M. A. Eslami, S. Mohamed, P. Battaglia, M. Jaderberg. and N. Heess, "Unsupervised learning of 3d structure from images," in Neural Inf. Process. Syst. (NeurIPS), 2016, pp. 4997-5005.
[40] S. M. A. Eslami, D. Jimenez Rezende, F. Besse, F. Viola, A. S. Morcos, M. Garnelo, A. Ruder- man, A. A. Rusu, I. Danihelka, K. Gregor, D. P. Reichert, L. Buesing, T. Weber, O. Vinyals, D. Rosenbaum, N. Rabinowitz, H. King, C. Hillier, M. Botvinick, D. Wierstra, K. Kavukcuoglu, and D. Hassabis, "Neural scene representation and rendering," Science, vol. 360, no. 6394, pp. 1704-1210 2018.
[41] R. Liu, J. Lehman, P. Molino, F. Such, E. Frank, A. Sergeev, and J. Yosinski, "An intriguing failing of convolutional neural networks and the CoordConv solution," in Neural Inf. Process. Syst. (NeurIPS), 2018 pp. 9605-9616.
[42] Civera, A. Davison, and J. Montiel, "Inverse depth parametrization for monocular slam," IEEE Trans. Robot., vol. 24, no. 5, 2008.
[43] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," " in Int. Conf. Learning Representations (ICLR), 2015.
10

---

Under review as a conference paper at ICLR 2020
[44] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, Song, H. Su et al., "Shapenet: An information-rict 3d model repository," arXiv preprint arXiv:1512.03012, 2015.
[45] J. Xiao, A. Owens, and A. Torralba, "Sun3D: A database of big spaces reconstructed using sfm and object labels," in Inl. Conf. Comput. Vis. (ICCV), 2013, pp. 1625-1632.
[46] Sturm, N. Engelhard, Endres, W. Burgard, and D. Cremers, "A benchmark for the evaluation of rgb-d slam systems," in IEEE/RSJ Int. Conf. Intell. Robot. Syst. (IROS). 2012, pp. 573-580.
[47] H. Hirschmuller, "Stereo processing by semiglobal matching and mutual information," " JEEE Trans. Pallern Anal. Machine Intell., vol. 30, 2, pp. 328-341 Feb. 2008
[48] N. Mayer, E. llg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox, "A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation," in Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2016, pp. 4040-4048.
[49] I. Laina, C. Rupprecht, V. Belagiannis, Tombari, and N. Navab, "Deeper depth prediction with fully convolutional residual networks," in 2016 Fourth international conference on 3D vision (3DV). IEEE, 2016, pp. 239-248.
[50] M. Pollefeys, R. Koch, and Van Gool, "Self-calibration and metric reconstruction inspite of varying and unknown intrinsic camera parameters," International Journal of Computer Vision, vol. 32, no. 1, pp. 7-25, 1999.
[51] I. Ronda, A. Valdés, and Gallego, "Line geometry and camera autocalibration," " Journal of Mathematical Imaging and Vision, vol. 32, no. 2, pp. 193-214, 2008.
[52] M. Hwangbo, J.-S. Kim, and T. Kanade, "Imu self-calibration using factorization," IEEE Transactions on Robotics, vol. 29, no. 2, pp. 493-507, 2013.
[53] C. Bailer, B. Taetz, and D. Stricker, "Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation," in IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2015, pp. 4023-4023.
11

---

Under review as a conference paper at ICLR 2020
6 APPENDIX
6.1 CONNECTION WITH TWO- VIEW TRIANGULATION
The problem of triangulation consists of computing the 3D coordinates of point given its (noisy) projections on two or more views and the camera parameters of the views. Hence, it is a geometric problem. Following the usual formalism of homogeneous coordinates [3], the perspective projection of a 3D point M on two cameras with projection matrices PI. P2 (comprising the intrinsic and extrinsic parameters of both views) given by Aim = Pi and 22m2 = P2M, where Å1, 22 are the projective scaling factors.
Given PI, P2; m,, m2, the linear triangulation algorithm 3 12.21, which tackles the triangulation problem in its most general setting (projective cameras), computes the 3D point M by minimizing the Rayleigh quotient [|AM|]/|]] || where A is the matrix
[malx P1 [m2)x P2
A m1, m2)
(3)
and [u]x is the cross product matrix (such that [u] V x v, for all v).
In case of multiple point correspondences {mi <+ m2) for I,..., the camera matrices Pi, P2 appear in the triangulation equations (3) of all of them, and hence, are "global" variables. If the camera matrices are known, then (i) every 3D point M' can be triangulated independently from the rest, and (ii) the triangulated point is a function of the point correspondences mi ++ m2.
M fmm m²; Pi. P2).
(4)
This intuition inspired the design of our modular architecture: First, a neural network regresses global variables which depend only on the two views, and then those global variables are used by local module to generate fully-convolutional net which transforms point correspondences (optical flow)
into depth.
6.2 TRAINING PROCESS
We train our model from scratch on the Scenesll dataset for approximately 150K steps using Adam optimizer with initial learning of Te 4. We normalize all losses with the number of points used to compute them. The loss weights for depth and smoothness Xp. A are 5.0 and 2.0 respectively. To increase generalization, x-axis and rotating them 180 degrees, both 50% probability. For fair comparison, we trained all baselines with exactly the same strategy and hyper-parameters on desktop PC equipped with an NVIDIA-GeForee 940MX.
perform data augmentation at training time by mirroring pairs on the
For the pose experiments in Sec. 4.4, we trained 2 hidden layer MLP with nodes and leaky ReLU activation function to predict relative camera motion between frames from the global parameters estimated by our global network. For all datasets and all variations, we trained with an T,, loss between estimated and real camera poses for 50K steps.
6.2.1 TUNING OF BASELINES
To fairly study the sparse training setting, we tuned the baseline to reach the best performance on the sparse training task. First, we changed the input layer of each baseline architecture. Exactly for ours, the baselines" input consists of the concatenation of the image pair and the optical flow on the last channel. Given the very sparse supervision signal, we noticed that the ReLu activation function generated extremely sparse and noisy gradients. Therefore, we modified the original activation function of DispNet [8). FCRN 49 and Figen (6 from Relu LeakyRelu. This change improved the performance of the baselines of up to 50% on average over metrics.
6.3 COMPARISON WITH STRUCTURE FROM MOTION METHODS
Dense Structure from Motion (SfM) methods 24
23
can recover the depth map of scene from two
more views using projective geometry 3 We now compare our global-loca local architecture two
12

---

Under review as a conference paper at ICLR 2020
SfM baselines proposed by Ummenhofer al. [2]] one that computes correspondences between images by matching SIFT keypoinnss (SIM-SIFT) and another that uses optical flow instead [53] (SfM-Flow). Given the correspondences, the essential matrix is estimated with the normalized 8-point algorithm and RANSAC 3 and further refined by minimizing the reprojection error with the ceres library. Finally, the depth maps computed by plane sweep stereo and optimized with the variational approach of Hirshmueller et al. [47]
Results in Table 6 show that our approach achieves on average 63% better error than SIM-SIFT and 43% better error than SfM-Flow over all datasets and metrics. Indeed, the considered datasets represent challenge for geometry- based methods given the presence of large homogeneous regions, occlusions, and small baselines between views, which are typical factors encountered in indoor scenes. In addition, geometry-basee methods are known to be subject to correspondence errors, which the aforementioned factors generally worsen. Nonetheless, our method remains competitive against the geometry- based techniques, outperforming them on two out of three metries. Furthermore, as we show in the next section, our method is much more robust to errors in the optical flow estimates than classic triangulation.
Dataset
Scenesll
SUN3D
RGB-D
Abs-Inv
Abs-Rel
S-RMSE
Abs-Inv
Abs-Rel
S-RMSE
Abs-Inv
Abs-Rel S-RMSE
Datasct-Mean
0.069
0.771
0.940
0.081
0.730
0.378
0.062
0.695
0,475
SIM-SIFT 23
0,051
1.027
0.900
0.029
0.286
0.290
0,050
0.703
0.577
SIM-Flow 25
0.038
0.776
0.793
0.029
0.297
0.284
0.045
0.613
0.548
Ours
0.033
0.43
0.61
0.045
0.48
0.23
0.040
0.44
0.33
Table 6: Comparison Structure from Motion (SfM) baselines. Our approach outperforms SIM methods on all datasets and metrics, except for two. This shows that learning from sparse supervision can be competitive with classic geometry- techniques.
6.4 EXPERIMENTS WITH GROUND TRUTH
The main intuition driving the design of our architecture consists of the fact that the relative pose between two images control the conversion of correspondences to depth. Since depth can be computed analytically given the correspondences (in non-degenerate cases), can also our model learn this relation when correspondences are perfect the relative pose between cameras is given? How does it compare to the triangulation equations in term of performance and ability to handle false correspondences?
To answer these questions, we trained our network with either perfect flow or given pose. As baselines, we used both the simple triangulation equation and the SfM-pipeline presented in Sec. 4.3, but provided with ground-truth camera motion. In addition, we also compare our approach to linear triangulation with the pose predicted by our global parameter network after supervised refinement on ground-truth poses (see Sec. 4.4). The results of this analysis are reported in Table
Dataset
Scenesl1
SUN3D
RGB-D D
Abs-Inv Abs-Rel S-RMSE Abs-Inv Abs-Rel Rel S-RMSE Abs-Inv Abs-Rel S-RMSE
Triang.-P. Pose
0.061
0.48
0.65
0.65
0.78
0.52
0.50
0.52
0.73
Triang.-GT Pose
0.020
0.25
0.48
0.14
0.33
0.44
0.091
0.36
0.49
SIM-GT Pose 25
0.023
0.35
0.62
0.020
0.22
0.24
0.026
0.34
0.398
Ours-GT Flow
0.015
0.24
0.46
0.026
0.26
0.20
0.040
0.35
0.32
Ours-GT Pose
0.021
0.28
0.54
0.038
0.37
0.32
0.043
0.44
0.38
Ours
0.033
0.43
0.61
0.045
0.48
0.23
0.040
0.44
0.33
Table 7: Comparison of our approach with different input modalities triangulation with perfect pose, triangulation with pose estimated by finetuning our global net as Sec. 4.4 (P. Pose), and the SfM pipeline with ground-truth pose. Although slightly outperformed by the SfM baseline with pose information, our approach is significantly better than naive triangulation the real datasets, where correspondences indicating that approach learns filter these correspondence emrrs. Providing ground-truth relative pose between images or perfect correspondences generally increases the network performance, showing ability of our model to learn the relationship between these two modalities in ideal conditions.
generally very noisy.
13

---

Under review as a conference paper at ICLR 2020
Scenes
SUN3D
RGB-D
Per Pincl Abc. lnv Depth Error
P Lutton Pee
le 101
$4
04
6.0
018
532 6.0
3.0 605
6.5
610
0.0
10
600 6.5
2.90
0.5
6.99
Nomalizzd Flow Error
Figure 5: Relation between per pixel flow and depth error our method and triangulation, either with perfect pose (GT Pose) or with the pose provided by our fine-tuned global network (see Sec. 4,4). Our approach learns to filter out errors in correspondences hy exploiting its receptive field larger than one and regularities of those errors in the data.
Performances on Scenesl show that, when the optical flow generated by PWCNet are very precise, naive triangulation performs very competitively, even outperformng the more complex SfM method. Our network perform comparably to this baseline, showing indeed its ability to learn the mathematical relation between flow and depth. However, triangulation very sensitive to its inputs: when the precision of the extrinsic parameters or of the correspondences decreases, its performance drastically drops. This can be observed from the results on the SUN3D and RGB-D datasets (Table 7. where the optical flow generated from PWCNet is significantly worse than the one on Scenes 1. In contrast, as we show in Fig. our network can cope against these inaccuracies, outperforming triangulation in the real datasets. However, our approach is still not competitive with the SfM pipeline with given extrinsic parameters on the SUN3D and RGB-D datasets: this is an indicator of the difficulty coming from estimating good global and local parameters when both training flows and depths noisy, but not of our network ability to learn the relationship between these two modalities. Indeed, when the network is provided with perfect correspondences, performance generally increases.
6.5 FINE-TUNING OPTICAL FLOW
For all experiments above, have always assumed pre-computed optical flow field is provided as input. This optical flow, generated by the off-the-shelf PWCNet architecture [16), was fixed throughout training. In this section, we study the case when also the parameters of PwCNet are fine-tuned during training with the sparse depth loss. Table & shows the results of this evaluation. Interestingly, finetuning the parameters of PWCNet performs worse than fixing them. Such finding can be explained by the fact that the sparse depth loss is not sufficient to train the large number of PWCNet parameters. Indeed, we noticed a decrease the training error of approximately 10%, indicating an over-fitting to the observed training points. As an additional baseline, we add to PWCNet a small convolutional head to convert flow to depth and train everything with the sparse depth loss. The convolutional head consists of three convolution with (32,16. number of filters, stride and filter size 3. The result of this approach, presented in the first row of Table confirms that the sparse loss does not provide enough feedback to fine-tune correspondences.
Dataset
Scenes
SUN3D
RGB-D
Abs- Inv Abs-Rel S-RMSE Abs- Inv Abs- Rel
S-RMSE
Abs-
Abs- S-RMSE
PWC (+)
0.046
0.63
0.81
0.081
0.87
0.37
0.047
0.51
0.42
Ours (finetune PWC)
0.038
0.48
0.70
0.042
0.40
0.25
0.046
0.41
0.35
Ours (no finetune)
0.033
0.43
0.61
0.045
0.48
0.23
0.040
0.44
0.33
Table 8: Fine-tuning the parameters of PWCNet with very sparse depth loss performs worse than fixing them. (*) Indicates that small convolutional head has been added to the PWCNet architecture.
14

---

Under review as a conference paper at ICLR 2020
6.6 QUALITATIVE RESULTS
We show more qualitative results of depth estimation on the test set of our evaluation datasets in Fig. 6 Fig- and Fig. g Despite being trained with very sparse supervision, our approach learns predict smooth depth maps with sharp edges, comparable to the ones an encoder-decoder architecture learns with dense supervision. In contrast, an encoder-decoder architecture fails to learn smooth depths when trained with sparse supervision.
Fig. shows some of the filters produced by the local network convert optical flow into depth. Since converting flow to depth depends on the relative transformation between the two views, those filters are uppt-dependent. Generally, filters are different for each image pair. However, when the relative transformation between the input views is similar, filters also tend to be similar (first and second row of Fig- 9) contrast, when the relative transformation between views is completely different, filters tend to acquire dissimilar pattern (first and third row of Fig.
Image
GT Depth
Struct2Depth
DispNet
Ours
Figure 6: Qualitative results on the simulated Scenesll I dataset. Given the availability of noise- free depth maps for training and high quality optical flow, our approach can learn extremely sharp depth maps, comparable to the ones learned by encoder-decoder with dense supervision.
Image
GT Depth
Struet2Depth
DispNet
Ours
Figure 7: Qualitative results SUN3D: Given the very noisy optical flow estimated by PWCNet and
presence of noise in the training depth maps, the performance of all methods drops in this dataset. However, our approach is still able to leam smooth depth maps. Interestingly, our model can pick details which are not present in the ground ground-truth depth (top-row).
15

---

Under review as a conference paper at ICLR 2020
Image
GT Depth
Struct2Depth
DispNet
Ours
Figure 8: Qualitative results RGB-D, Also this dataset represents challenge for methods, given large baseline between views, noisy correspondences and noisy training depth maps. Nonetheless, our approach still able to estimate sharp depth maps, sometimes capturing fine details which even an cncoder-decoder trained with dense supervision fails to catch (bottom-row).
16

---

Under review as a conference paper at ICLR 2020
Image Pair and Local Network Filters
Figure 9: Local network filters generated by several image pairs. Generally, filters different for each image pair. However, when the relative transformation between the two views similar, filters also tend to be similar (first and second row). In contrast, when the relative transformation is completely different, filters tend to have dissimilar pattern (first and third row).
17


Reviews:
[{"review_id": "Skgsjr36Kr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a novel global-local network, which can be trained with extremely sparse ground truth, to predict dense depth. Though widely applied on the task of segmentation, the use of only uncalibrated input and extremely sparse label for depth estimation is novel. By incorporating optical flow and decoupling global and local modules, this pipeline aligns well with disparity-based geometric methods. The paper is generally well-written and easy to follow. \n\n\nHere are some concerns:\n\nSome important details are missing in order to reproduce the results. For example, the forward pipelines of the global module and the local module are covered with only few sentences. The authors should expand the section of methodology and give better formulations of their pipeline.\n\nHow much does the quality of input optical flow affect the results? Also, the author claimed ‘the network has the opportunity to correct for small inaccuracies or outliners in the optical flow input’, could you show any result related to this claim? (e.g. flawed optical flow but good output)\n\nCan you try to incorporate the optical flow module and fine-tune for better results? \n\nAre there some failure cases, and some visualizations on the filter banks of the local network?\n\n=========================================================\nAfter Rebuttal:\n\nI thank the author for the response. \n\nSince there is a possibly more realistic setting that we train the model with dense depth annotations we currently have and use very sparse annotations to adaptive to new environments where we are hard to gain dense annotations. Also, the optical flow module relies on extra annotations to train. I will keep my origin scores.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1ldszjk9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a depth learning architecture (global-local structure) from two images. It claims that SoTA depth can be estimated from the supervision of a very sparse ground truth by leveraging the optical flow information between two images. In the experiments, it shows superior performance than other baseline methods such as Eigen's network and DispNet. \n\nPros:\n1: The paper is well written and motivations are clearly explained. \n2: The architecture proposed is reasonable and generate good results, since it accept the ground truth scale from sparse map and relative dense matching cues from optical flow, where implicitly relative camera pose is from global module. \n\n\nCons:\n1`: It is a fairly standard network design similar to DeMoN[25], where motion network for global pose and local dense network for local matching.\n\n1: The claim of  robustness to camera intrinsics is not solved in principle but due to training using ground truth from multiple dataset . It still suffer from depth motion confusion if there is no ground truth depth guidance when testing.  It is also unfair for comparison of this metric with unsupervised approach where a universal intrinsic is assumed. \n\n2: I think the comparison might not be fair since the baselines are all single image estimation networks, while the approach has two images, where disparities are serving as a strong cue for depth. Other possible  architectures such as flow net, pwc net,  DeMoN[25] and stereo networks such as (gc-net or psm-net) should be considered since these networks are more focus on feature matching.  \n\n3:  Even for single image network, eigen's method is not SoTA, the author may consider (1)  as one of the baseline, etc.\n\n(1) \"Deeper Depth Prediction with Fully Convolutional Residual Networks\"\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1l1jzoOtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for learning to predict a depth map given a pair of images. Training is done is an unsupervised way except for a small set of image locations for which the absolute 3D locations are known. The optical flow between the 2 images is computed automatically. A network is trained to predict the camera motion between the 2 images from the images and the optical flow. From this global motion, convolutional filters are predicted, in order to transform the optical flow into a depth map.\n\nThe use of image locations with known 3D locations is motivated in the paper to \"simulate\" the knowledge of depth coming from a haptic system.\n\nI have several concerns about this paper.\n\n* My main concern is that the paper compares against only a few papers (1 from 2014, 1 from  2016, and one more recent from 2019) while the literature is extremely vast, and visually, the results seem far from the state-of-the art. See for example:\n\nHuangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian Reid. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In CVPR, 2018.\n\nAnurag Ranjan, Varun Jampani, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black. Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. In CVPR, 2019.\n\nChaoyang Wang, Jose Miguel Buenaposada, Rui Zhu, and Simon Lucey. Learning depth from monocular videos using\ndirect methods. In CVPR, 2018.\n\nIn fact, these papers have simpler requirements as they do not need 2 images at run-time, nor 3d data at training time. I thought for a moment that the advantage of the paper was to be able to predict an absolute motion and an absolute depth (this is not impossible if the method is able to estimate the scale from known objects, as it is suggested from the introduction). This was however incorrect as the text says in the middle of Section 4.1 \" we resolve the inherent scale ambiguity by normalizing the depth values such that the norm of the translation vector between the two views is equal to 1\".\n\n* The network used to predict optical flow was trained with a large amount of supervised data. As image matching is the most difficult task, it is difficult to claim that the method is unsupervised.\n\n* The motivation for having sparse measurements is to \"simulate\" haptic. This is fine for me in principle, however haptic measurements would probably have large errors, while it seems that the experiments use ground truth values for these points.\n\nOne minor remark:  End of Section  4.4: What is a \"179% error reduction\"? How is computed the percentage?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a deep network architecture for learning to predict depth from images with sparsely depth-labeled pixels. 

This paper was subject to some discussion, since the authors felt that the approach was interesting and the problem-well motivated. Some of the concerns about experimental evaluation (especially from R1) were resolved due the author's rebuttal, but ultimately the reviewers felt the paper was not yet ready for publication. 
Decision: Reject