Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper pushes forward the research of deep learning based video 3D reconstruction by decomposing the problem in two-stages:
1. Depth estimation from multi-view stereo
2. Camera pose estimation from optical flow estimation and PnP SE3 pose,
which turns out to achieve state-of-the-art performance on public datasets. 

In general, the iterative procedure of this paper is similar to DeepTAM (Zhou et al., 2018), the major difference is that the camera pose is estimated by PnP with estimated Flow but not directly predicted from sub-network structures, which contributes to the higher tracking accuracy and further improves the accuracy of multi-view stereo depth estimation. 

A multi-view camera pose estimation (the global pose optimization in the paper) is also proposed to utilize the relative relation between all video frame pairs, which is difficult for fully CNN based pipeline and maximizes the use of the explicit PnP optimization. 

After that, a residual flow field is predicted using CNNs and camera poses is further refined by minimizing the geometric re-projection error.

So this paper focuses more on the camera pose estimation than the depth, which is a good start point to achieve better multi-view capabilities in a CNN framework.


However, I still have several concerns for this paper:

1. The full system is highly engineered and complicated. For example, the feature map is extracted from two hour-glass networks, which seems over-complicated for feature extraction, and the multi-view stereo network using four 3D hour-glass networks, which consumes a large amount of memories. So I would like to see the authors demonstrate the performance from in simpler settings, e.g. the feature map can be a single encoder-decoder and the multi-view stereo is done by correlation and 2D convolution. So I would like to see the inference time, peak memory consumption and the model size. Ablation studies will also help the readers to understand whether performance gain is from the pose estimation or the network capacities, which is unclear in the current paper (Appendix.D). 

2. The comparison with state-of-the-art conventional system is missing. For example, is the camera pose estimation better than initializing the DSO (Engel et al., 2018) with a monocular depth estimation? In real applications, if the performance gain is insignificant, the conventional method will still be a better choice because the CNN based methods are computationally expensive on platforms without powerful GPUs. I will not downgrade the rating if the performance gain is insignificant, but it is necessary to see the comparison.


3. Even the difference is ignorable for performance, I hope the author could use adjoint when deriving the derivatives in Eq.(11) of Appendix A.1. The author can refer to this tutorial http://ethaneade.com/lie.pdf or the text book https://www.eecis.udel.edu/~cer/arv/readings/old_mkss.pdf.

PS: For the current version with 10 pages, I lean to a borderline score, but I select 6 because 5 is not an option. I will keep or raise the current score if my concerns are addressed during rebuttal. 
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper proposes a framework for training machine learning models that simultaneously estimates depth of objects and poses of a single camera in a sequence of images from a single camera, in others words, a video.
In a video, to estimate depth of objects, which is a main objective of this paper, we need to know the relative position and rotation information of a single camera in a sequence of images in a video (motional information). However, to estimate the relative position and rotation information of the camera, we need to know a ground truth depth information of each objects in each image.
The main idea works just like EM or alternating optimization. The depth module estimates depth of objects in a sequence of images assuming the relative position and rotation information of a single camera is given. The motion module estimates motional information of a camera assuming the depth of each object is given.
The authors formulate the aforementioned two modules as neural networks, so that they can be trained end-to-end, and proposes various way of initializing the two modules.

While the idea is simple, I think the paper is well-written and the experiments show a superior performance over existing approaches.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The authors proposed to estimate depth from a video sequence. In the model, the pipeline iteratively estimates motion and depth by separate modules, which can be trained in an end-to-end fashion. The numerical results show better scores than the other SOTA methods.  Overall, I think this is a useful work and may be considered for publishing.

The following are the detailed comments:
1. The introduction of the related works is well written.
2. The empirical comparison is quite thorough and demonstrates the proposed method using sequence and multi-frame is very useful.
3. The method has good generalization.

4. Besides the empirical results, I found the paper reads quite mechanical and provides very limited understanding if any. It would be much better if the authors can make further effort to understand where exactly the performance comes from. With a more complete story, the paper would have a better potential to last longer.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This work proposes a neural network architecture for joint depth and camera motion estimation on video sequences. The authors propose an architecture that incorporates classic principles from SfM, namely depth computation based on cost volumes and motion estimation based on the reprojection error of features. Extensive experiments on a variety of datasets are shown and support that this approach provides strong results.

+ Principled approach that marries the best aspects of deep learning with classic principles from multi-view geometry.

+ Well-written paper

+ Generalizes well

+ Significantly outperforms the state of the art

+ Clearly shows that more views help

+ Seems to be robust to initialization


Question: What happens if more Gauss-Newton steps are made? Could one trade computation for quality here?


Minor issues:

- Equation (1), right: f_z should probably be f_x
- Paragraph between Eq. (1) and (2): There seems to be something wrong with the typesetting of x^i=... (e.g. the equals sign)
- Same for the paragraph before Eq. (1) and x=(u,v)


Summary: This paper presents a well-engineered and non-trivial system that leverages principles from different fields in a very reasonable way. The results look great both qualitatively and quantitatively. The experiments are extensive and show a clear improvement over the state-of-the-art.

review_title: Official Blind Review #2

Meta-review: This work proposes a CNN architecture for joint depth and camera motion estimation from videos. The paper presents a differentiable formulation of the problem to allow its end-to-end learning, and the reviewers unanimously find the proposed approach reasonable and agree that this is a solid paper. Some of the reviewers find the method itself to be too mechanical, but they all agree that this is a well-engineered solution.
Decision: Accept (Poster)