Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary:

The authors propose a SfM model which integrates geometric consistency with a learned pose and depth network. An initial estimate of depth and pose are used to construct pose and depth cost volumes, which are then fed into a pose regression and depth refinement network, to produce a new set of cost volumes, and so on. In this manner, the pose and depth estimation are improved iteratively.

Strengths:

The proposed model is well motivated and shows strong performance and generalization ability on several datasets. There are convincing experiments to show the importance of the P-CV network.

Weaknesses:

The authors claim that the LM optimization in BA-Net is memory inefficient and may lead to non-optimal solutions. It’s not clear to me that the proposed method can guarantee optimality any better. It’s also unclear if the proposed method is more memory efficient, since the authors only unroll 4 iterations of it.

Other comments:

It would be very interesting to see the test time behavior of the network when it is run with more iterations than it is trained with (say 10 or 20), especially since the depth error does not seem to have stopped decreasing at only 4 iterations.

It’s not made entirely clear whether the training backpropagates through the update/construction of the pose and depth cost volumes. 

In equation 5, “x” should be “i”.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors propose a physical driven architecture of DeepSFM to infer the structures from motion. Extensive experiments on various datasets show that the model achieves the state-of-the-art performance on both depth and pose estimation. In general, the paper is clearly written but I still have several concerns. 
1.	The paper is easy to follow but the authors are expected to clarify the rationality in integration of the loss function. How the parameter of \lambda_r, \lambda_t, and \lambda_r influence the performance. It would be better if the authors could present some analysis. 
2.	The experiments are rather insufficient. The authors are expected to make more comprehensive analysis with the state-of-the-art methods, and also analyze why some alternative methods outperforms the proposed methods in table I and table II. 
3.	The experiments in section 4.3 are also expected to be improved. It is difficult to draw a conclusion that the method is better than other ones based on such limited experiments.  

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper tackles Structure from Motion, one of the canonical problems in computer vision, and proposes an approach that brings together geometry and physics on one hand and deep networks on the other hand. Camera unprojection and warping (of depth maps and features) are used to build a cost volume onto hypothetical planes perpendicular to the camera axis. Similarly, various camera poses are sampled around an initial guess. A deep network regresses form the cost volume to a camera pose and a depth map. The method can be applied iteratively, using the outputs of the current stage as the initial guess of the next one. Training is supervised, and the the results are evaluated on multiple datasets.

I am inclined to recommend accepting the paper for publication, because it addresses a canonical problem, outperforms the state of the art on multiple datasets and brings together geometry / physics and deep learning, which is IMO very a promising and underexplored direction.

I found the method section a bit difficult to read though, and even after several readings I cannot get my head around it. Specifically, here are some issues that I hope the Authors could clarify.

1. In Sec. 3 the Authors write "We then sample the solution space for depth and pose respectively around their initialization". However in Sec 3.2 they write "we uniformly sample a set of L virtual planes {dl} Ll=1 in the inverse-depth space". In what way are the planes "around their initialization"? If the initial depth map spans over multiple orders of magnitude, will the planes be uniformly sampled between the minimum and maximum disparity of the initial map? If yes, it seems that the initial depth map is not really needed, just its minimum and maximum value is needed, but then how come the method can be applied iteratively with respect to depth?

2. The Authors mention that depth maps are warped onto the virtual planes using differentiable bilinear interpolation. Is there a mechanism to protect from interpolating across discontinuities? If no, were bleeding edge artifacts observed?

3. In the introduction, the Authors point that prior methods have trouble dealing with textureless, reflective or transparent approaches, but it's not clear form the paper where it addresses these cases, and if yes, what is the mechanism for that.

Lastly, if the authors are not planning to release the code, the implementation details section is a bit too high-level and does not contain enough details to reimplement the Author's technique. For example, "our network learns a cost volume of size L × W × H using several 3D convolutional layers with kernel size 3 × 3 × 3"  - more details about this network are needed, as well as the others in the paper.


review_title: Official Blind Review #3

Meta-review: Main content:  Physical driven architecture of DeepSFM to infer the structures from motion
Discussion:
reviewer 1: well-motivated model with good solid experimental results. not clear about the LM optimization in BA-Net is memory inefficient 
reviewer 2: main issue is the experiments could be improved.
reviewer 3: well written but again experimental section is lacking
Recommendation: Good paper and results, but all 3 reviewers agree experiments could be improved. Rejection is recommended.
Decision: Reject