Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper provides a depth learning architecture (global-local structure) from two images. It claims that SoTA depth can be estimated from the supervision of a very sparse ground truth by leveraging the optical flow information between two images. In the experiments, it shows superior performance than other baseline methods such as Eigen's network and DispNet. 

Pros:
1: The paper is well written and motivations are clearly explained. 
2: The architecture proposed is reasonable and generate good results, since it accept the ground truth scale from sparse map and relative dense matching cues from optical flow, where implicitly relative camera pose is from global module. 


Cons:
1`: It is a fairly standard network design similar to DeMoN[25], where motion network for global pose and local dense network for local matching.

1: The claim of  robustness to camera intrinsics is not solved in principle but due to training using ground truth from multiple dataset . It still suffer from depth motion confusion if there is no ground truth depth guidance when testing.  It is also unfair for comparison of this metric with unsupervised approach where a universal intrinsic is assumed. 

2: I think the comparison might not be fair since the baselines are all single image estimation networks, while the approach has two images, where disparities are serving as a strong cue for depth. Other possible  architectures such as flow net, pwc net,  DeMoN[25] and stereo networks such as (gc-net or psm-net) should be considered since these networks are more focus on feature matching.  

3:  Even for single image network, eigen's method is not SoTA, the author may consider (1)  as one of the baseline, etc.

(1) "Deeper Depth Prediction with Fully Convolutional Residual Networks"


review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposed a novel global-local network, which can be trained with extremely sparse ground truth, to predict dense depth. Though widely applied on the task of segmentation, the use of only uncalibrated input and extremely sparse label for depth estimation is novel. By incorporating optical flow and decoupling global and local modules, this pipeline aligns well with disparity-based geometric methods. The paper is generally well-written and easy to follow. 


Here are some concerns:

Some important details are missing in order to reproduce the results. For example, the forward pipelines of the global module and the local module are covered with only few sentences. The authors should expand the section of methodology and give better formulations of their pipeline.

How much does the quality of input optical flow affect the results? Also, the author claimed ‘the network has the opportunity to correct for small inaccuracies or outliners in the optical flow input’, could you show any result related to this claim? (e.g. flawed optical flow but good output)

Can you try to incorporate the optical flow module and fine-tune for better results? 

Are there some failure cases, and some visualizations on the filter banks of the local network?

=========================================================
After Rebuttal:

I thank the author for the response. 

Since there is a possibly more realistic setting that we train the model with dense depth annotations we currently have and use very sparse annotations to adaptive to new environments where we are hard to gain dense annotations. Also, the optical flow module relies on extra annotations to train. I will keep my origin scores.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper presents a method for learning to predict a depth map given a pair of images. Training is done is an unsupervised way except for a small set of image locations for which the absolute 3D locations are known. The optical flow between the 2 images is computed automatically. A network is trained to predict the camera motion between the 2 images from the images and the optical flow. From this global motion, convolutional filters are predicted, in order to transform the optical flow into a depth map.

The use of image locations with known 3D locations is motivated in the paper to "simulate" the knowledge of depth coming from a haptic system.

I have several concerns about this paper.

* My main concern is that the paper compares against only a few papers (1 from 2014, 1 from  2016, and one more recent from 2019) while the literature is extremely vast, and visually, the results seem far from the state-of-the art. See for example:

Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian Reid. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In CVPR, 2018.

Anurag Ranjan, Varun Jampani, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black. Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. In CVPR, 2019.

Chaoyang Wang, Jose Miguel Buenaposada, Rui Zhu, and Simon Lucey. Learning depth from monocular videos using
direct methods. In CVPR, 2018.

In fact, these papers have simpler requirements as they do not need 2 images at run-time, nor 3d data at training time. I thought for a moment that the advantage of the paper was to be able to predict an absolute motion and an absolute depth (this is not impossible if the method is able to estimate the scale from known objects, as it is suggested from the introduction). This was however incorrect as the text says in the middle of Section 4.1 " we resolve the inherent scale ambiguity by normalizing the depth values such that the norm of the translation vector between the two views is equal to 1".

* The network used to predict optical flow was trained with a large amount of supervised data. As image matching is the most difficult task, it is difficult to claim that the method is unsupervised.

* The motivation for having sparse measurements is to "simulate" haptic. This is fine for me in principle, however haptic measurements would probably have large errors, while it seems that the experiments use ground truth values for these points.

One minor remark:  End of Section  4.4: What is a "179% error reduction"? How is computed the percentage?

review_title: Official Blind Review #1

Meta-review: This paper proposes a deep network architecture for learning to predict depth from images with sparsely depth-labeled pixels. 

This paper was subject to some discussion, since the authors felt that the approach was interesting and the problem-well motivated. Some of the concerns about experimental evaluation (especially from R1) were resolved due the author's rebuttal, but ultimately the reviewers felt the paper was not yet ready for publication. 
Decision: Reject