Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The underlying problem considered in this manuscript is inferring depth from geometry of two dimensional images. The novelty here is to integrate a semantic classification model along with depth inference. It is a challenging problem and a neat idea to pursue. The paper is well-written and easy to follow. 
However, the empirical work in the paper is not persuasive. Table 1 contains results whose significance is hard to judge. What does a RMSE difference of 2.3 mean in the context of depth estimation? Table 1 carries no uncertainty in results which is just not acceptable in a setting that has several sources of uncertainty. Similarly, Fig 4 shows there is advantage in the use of the pre-trained semantic network, it is not clear if this difference is significant. And I also think consideration should be given to the fact that in a deployment setting, new objects not previously seen in the semantic categories (UFO) may appear and one ought to understand if the semantic network might decrease performance (because of the unseen class). Hence I think which the idea advanced in the paper has merits, the manuscript is not really ready for publication.  
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposes a using pixel-adaptive convolutions to leverage semantic labels in self-supervised monocular depth estimation. The semantic features are predicted by a pretrained network rather than relying on a ground truth. Moreover, a two-stage training process in proposed in order to filter out images leading to erroneous SfM predictions. The method is evaluated with different networks on the KITTY dataset.

The paper is very well written and clear. The applications of per-pixel convolutions to this problem seems sound and the experimental validation seems satisfactory. I have however one main concern (1) and a few additional questions below:

1) While (Guizilini 2019) shows that using a larger set of unannotated videos and allows the self-supervised method to eventually outperform supervised methods, this study is not done here. This makes me question the applicability of the approach, as using large unlabelled videos would probably lead to noisy segmentations that could be unhelpful to the depth estimation. Showing an improvement over the supervised baseline would be a much stronger experimental validation, as for now it is difficult to know exactly why in which scenario this method should be used, rather than a supervised network or vanilla packnet.

2) I see that you obtain the same numbers in Table 2 / PackNet / row 1 as in (Guizilini 2019); I would like to confirm that you used exactly their self-objective loss, in all your experiments? I would suggest adding to section 3.1. the fact the fact that the loss is the same is in (Guizlini 2019), as a reader could assume that there is novelty in the loss formulation.

3) Have you tried fine-tuning the whole architecture including the semantic network end-to-end?

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This work proposes to leverage a pre-trained semantic segmentation network to learn semantically adaptive filters for self-supervised monocular depth estimation. Additionally, a simple two-stage training heuristic is proposed to improve depth estimation performance for dynamic objects that move in a way that induces small apparent motion and thus are projected to infinite depth values when used in an SfM-based supervision framework. Experimental results are shown on the KITTI benchmark, where the approach improves upon the state-of-the-art.

Overview:

+ Good results
+ Doesn't require semantic segmentation ground truth in the monodepth training set

- Not clear if semantic segmentation is needed
- Specific to street scenes
- Experiments only on KITTI

The qualitative results look great and the experiments show that semantic guidance improves quantitative performance by a non-trivial factor. The qualitative results suggest that the results produced with semantic guidance are sharper and more detailed. However, it is not clear that using features from a pre-trained semantic segmentation network is necessary. The proposed technical approach is to use the pixel-adaptive convolutions by Su et. al. to learn content-adaptive filters that are conditioned on the features of the pre-trained semantic segmentation network. These filters could in principle be directly learned from the input images, without needing to first train a semantic segmentation network. The original work by Su et. al. achieved higher detail compared to their baseline by just training the guidance network jointly.  Alternatively, the guidance network could in principle be pre-trained for any other task. The main advantage of the proposed scheme is that the guidance path doesn't need to be trained together with the depth network. On the other hand, unless shown otherwise, we have to assume that the network needs to be pre-trained on some data that is sufficiently close to the indented application domain. This would limit the approach to situations where a reasonable pre-trained semantic segmentation network is available.

The proposed heuristic to filter some dynamic objects is very specific to street scenes and to some degree even to the KITTI dataset. It requires a dominant ground plane and is only able to detect a small subset of dynamic motion (e.g. apparent motion close to zero and object below the horizon). It is also not clear what the actual impact of this procedure is. Section 5.4.2 mentions that Abs. Rel decreases from 0.121 to 0.119, but it is not clear to what this needs to be compared to as there is no baseline in any of the other tables with an Abs. Rel of 0.121. Additionally, while the authors call this a minor decrease, the order of magnitude is comparable to the decrease in error that this method shows over the state-of-the-art (which the authors call statistically significant) and also over the baselines (c.f. Table 2). Can the authors clarify this?

Related to being specific to street scenes: The paper shows experiments only on the KITTI dataset. The apparent requirement to have a reasonable semantic segmentation model available, make it important to evaluate also in other settings (for example on an indoor dataset like NYU) to show that the approach works beyond street scenes (which is one of the in practice not so interesting settings for monocular depth estimation since it is rather easy to just equip cars with additional cameras to solve the depth estimation problem).

Need for a reasonable segmentation model: It is not clear in how far the quality of the segmentation network impacts the quality for the depth estimation task. What about the domain shift where the segmentation model doesn't do so well? Even if the segmentation result is not used directly, the features will still shift. How much would depth performance suffer?

Summary:
While the results look good on a single dataset, I have doubts both about the generality of the proposed approach as well as the need for the specific technical contribution.

=== Post rebuttal update ===
The authors have addressed many of my initial concerns and provided valuable additional experimental evaluations. While I'd like to upgrade my recommendation to weak accept, I strongly encourage the authors to provide additional experiments on different datasets (at least NYU). 
review_title: Official Blind Review #3

Meta-review: The paper proposes a using pixel-adaptive convolutions to leverage semantic labels in self-supervised monocular depth estimation. Although there were initial concerns of the reviewers regarding the technical details and limited experiments, the authors responded reasonably to the issues raised by the reviewers. Reviewer2, who gave a weak reject rating, did not provide any answer to the authors comments. We do not see any major flaws to reject this paper.
Decision: Accept (Poster)