Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presents a method for semantic-guided self-supervised depth estimation from monocular images. They propose semantic guidance to improve depth estimation performance. This is obtained by applying semantic guidance at multiple levels in the decoder via an attention layer and via feature enhancement in edges areas of the image. Experiments show the proposed method reaches state of the art performance for self-supervised monocular depth estimation in structure-from-motion setting.

Pros
- given semantic guidance, this method seems well designed and effective to enhance depth estimation performance with a relatively low computation overhead in order to provide semantic guidance
- interesting semantic guided edge point sampling strategy
- the paper shows an improved semantic guidance with respect to Guinzilini et al 2020b

Cons
- not clear how the decoder was designed, why on some layer there is attention and on other concatenation + convolution? Is it experimentally obtained, a design choice? Could the authors give some explanation about it?
- personally I find difficult to understand the choice of parameters \mu, c and N related to the semantic guided edge point sampling strategy, could the authors provide some detail about it?
- about the point feature extraction and enhancement, two unclear parts for me. First, "we select the 0 − 2 layers of features" could you explain what features/feature blocks you are referring to? Maybe a small explanation could be useful or adding in Fig. 1 some corresponding number to relate to. Second, "point feature enhancement module, which merges and enhances these point representations via a set of 1-D convolutions", the PFE is not clear, set of 1-D convolution means that for each channel of the feature map there is a convolution?
- Eq. 8, i \in [2,4] is related to the darker box around the decoder blocks in Fig. 1?
- the training set for Cityscape is the train split with fine annotations or also with coarse annotations?

Overall, the paper is well written, the results are sound and minor improvements could make it worth acceptance.

review_title: The paper presents a method for semantic-guided self-supervised depth estimation from monocular images. Attention and edge feature enhancement are the key contributions. Overall, good paper some minor clarifications required.

final_score: rating: 7: Good paper, accept
review_content: The authors tackle the problem of self-supervised depth estimation and particularly address the issue of poor depth estimation on object boundaries. The introduction motivates the problem well, and the related work covers most of the relevant papers. The authors propose two main modifications allowing them to leverage an off-the-shelf semantic segmentation network: SEEM (semantic guided edge enhancement module) and a multi-level self-attention mechanism that fuses depth and semantic features at different levels. The combination of these contributions along with an off-the-shelf semantic segmentation network achieves impressive results, especially on the  boundaries of objects. 

Overall the paper is well written and the method clearly explained; the 2 diagrams in Figures 1 and 2 are very well done and explain the workings of the system quite well. The authors ablate their contributions clearly showing their respective impact on the overall performance. The evaluation on KITTI shows that the method compares favorably with related work. The presentation could be improved by addressing the points mentioned below, and specifically: (i) it would be interesting to see how the quality of the off-the-shelf semantic segmentation network affects the depth results (see below) and (ii) it would be nice to see a per semantic class evaluation (similar to Fig 4 in  (Guizilini et al., 2020b)) showing specific improvements after using the proposed modifications. 

Comments/suggestions for improvement:

* Does the quality of the off-the-shelf segmentation affect the result? There is an implicit assumption here that the pretrained, off-the-shelf network transfers well to the current domain. It would be interesting to explore how this affects depth estimation performance. 
* What data is the pretrained semantic segmentation network of Zhu et al. (2019) trained on? Could the authors report the performance of the network on the original training dataset, for reference? Guizilini et al. (2020b) report that their pre-trained semantic segmentation network achieves an mIoU of 75% on the Cityscapes val set; how does that compare to the network of Zhu et al. (2019) used by the authors?
* Table 1 - the numbers reported for Guizilini et al. (2020b) don’t seem to include the PackNet numbers reported in that paper (i.e. the abs_rel reported for 640x192 seems to be 0.102); while that network architecture is different, I would argue it’s still relevant as a comparison with state-of-the-art methods
* Table 3 in the supplementary is missing the Guizilini et al. (2020a) numbers on the KITTI improved ground truth.
* The paper is missing some details regarding the network architectures used - the authors should provide this information if possible, either in the main text or in the supplementary
* It would also be interesting to see the performance of the method with using full resolution images - would using semantic segmentation further help improve results in that setting?

Minor comments:
* Table 1 caption “D” denotes depth supervision - since there is no method using Depth supervision in the table this can be removed
* It would be nice if the authors could add F_E, F_D and F_S on Fig 1 to make it easier to follow the description in Sec 3.2
* Setting i \in [2,4] means using the self attention twice? Fig 1 has 3 self-attention blocks.

Post rebuttal: I thank the authors for their detailed response and paper revision. My concerns have been addressed, and I particularly appreciate the experiments presented Table 3, Figure 5 and Section 5.4. I am happy to maintain my rating and recommend acceptance. 
review_title: Official Blind Review #3

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This paper proposed a novel framework to improve self-supervised monocular depth estimation leveraging semantic features at local and global level. The proposed framework includes a semantic-guided edge enhancement module to extract and enhance point-based features around semantic boundaries. The proposed framework also incorporates feature fusion through a self-attention module at different feature levels for global fusion. Evaluation on KITTI datasets compared with recent state-of-the-arts are provided. The proposed method outperformed other monocular depth estimation methods including the ones that also leverage semantic information.

--------------------------------------------------------------------------------------------

Pros: 
* The paper is well formatted and easy to read. 
* The idea of semantic guidance both locally and globally is novel and interesting.
* The new framework performs the previous state-of-the-arts under the same network settings. 

Cons:
* The cityscapes experiment provides very limited contribution to the paper as only qualitative results are provided.
* The prediction accuracy of semantic segmentation branches is missing. Even Though it is trained with pseudo labels, it is still important to provide this result in order to justify that the semantic branch is providing reasonable predictions. 
* There is some other confusing description in the experiment section that raises concerns:
  - The paper keeps mentioning that the semantic pseudo labels are generated from off-the-shield algorithms which require no groundtruth. This argument is confusing as the pretrained model from reference work did use ground truth. 
  -The authors mentioned semantic labels for the test set of KITTI and Cityscapes. Why do you need to use these labels for testing when no semantic quantitative result is provided? Please provide more details there. Or one might assume the label is needed during network inference. 
  - The author mentioned that the final model is selected via validation set. Does that mean the reported number is not from a model converged with the training protocol? Or is it selected from multiple runs with random seed? Please provide more details there. 

----------------------------------------------------------
Other detail comments:

- In related work, self-supervised depth estimation, I think it would be good to mention the effort of network architecture improvement for depth estimation (E.g. Guizilini 2019a). Then emphasize that the experiment comparison rules out the impact of different network architecture in either related work or experiment section, given that the main result in Guizelini 2019b using another network is not compared in the table.
- Since both depth branches and semantic branches are trained together, how do you weigh the losses? It is a simple sum up or weighted? Please provide the detail for reproducibility.
- In figure 4. Is the semantic guidance map the groundtruth label or predicted maps?
- The author mentioned in Appendix C that binary semantic categorization is used. Please provide details on what classes are considered as foreground. 
- The description “insufficient depth representation” is a bit confusing. The solution to “insufficient” should be “providing more”, while the paper is improving and augmenting the representation. I would recommend the author consider changing the description here. 
- It would be nice to include some qualitative results along with the ablation analysis. 
--------------------------------------------------------
post-rebuttal:
I thank the authors for their clarification and ablation analysis. Most of my concerns are resolved. I will keep my original score and I think the paper could be accepted. 

review_title: Official Blind Review #2

final_score: rating: 5: Marginally below acceptance threshold
review_content: Strengths:

1. Writing is good. It is easy to read.
2. Although limited, the proposed term really improves the performance.
3. The qualitative results show better depth estimation in object boundary areas. 



Weakness:

4. “How to generate semantic labels” is a very important problem in this paper. Simply referring to (Zhu et al., 2019) is not sufficient, and I suggest authors at least using 3-5 sentences to describe that, covering introduction for the input, output, and methodology.

5. After reading (Zhu et al., 2019), I find that the ground-truth semantic labels are still required, and their method can be regarded as a data augmentation. Then I have serious concerns about semantic supervisions:
a. The ground truth for semantics requires human labelling, while the depth supervision needs LiDAR or Kinect-like sensors. In my opinion, the former one is more expensive than the latter, so the requirement of training the proposed model is higher than existing supervised methods.
b. By comparing the results, I find that the proposed method is not as good as supervised methods or self-supervised methods that use stereo pairs for training. 
c. Due to the requirement for ground truth semantics, the claim of self-supervision is not convincing.
d. It loses the advantage of self-supervised depth estimation. For example, existing depth CNN (Gordon et, al, iccv 2019) can be trained on a wide range of YouTube videos, where no semantic labels are provided. Although the pre-trained segmentation network could be used, their generalization is not validated.

6. The proposed method leads to very limited performance improvement, e.g., final results (absrel=0.105) vs baseline results (absrel=0.110) as shown in the Table 2, while it requires additional semantic labels for training. Considering the trade-off between performance improvement and requirement for training data, I have serious concerns about the contribution of this paper. 

7. Object boundary is hard to estimate in all dense prediction tasks. This paper improves that in depth estimation by using the well-labeled semantic maps. However, in practice, obtaining accurate semantic labels is difficult, and this is also a challlenging problem for semantic segmentation. 

8. The importance of depth sharpness on object boundary should be more clearly discussed, and the trade-off between such improvement and disadvantages caused by proposed solutions (as I mentioned above) should be discussed.

post-rebuttal: Authors pay much effort on addressing issues that I mentioned, and I appreciate that very much. My issues can be partially resoved, but the main issue is regarding the high-level thinking on semantic supervisions, which is impossible to be fully addressed in a rebuttal. Besides, I agree with AC that (a) the improvement is very limited and (b) semantic labels are hard to obtain. Overall, I raise my rating to 5 (not so good, but could be accepted), because I really appreciate authors' effort.



review_title: This paper proposes to use semantic labels for improving the self-supervised depth estimation. The results show better depth estimation in object boundary areas. However, I have serious concerns about the semantic supervision, the claim of self-supervised learning, the trade-off between performance improvement and training data requirement, and the practicality of proposed method.

Meta-review: The authors address the problem of self-supervised monocular depth estimation via training with only monocular videos. They propose to use additional information extracted from semantic segmentation at training time to (i) provide additional “semantic context” supervision and (ii) to improve depth estimation at discontinuities through an edge guided point sampling based approach. Results are presented on the KITTI and Cityscapes datasets.  

One of the main concerns is related to the utility of the semantic supervision given the relative cost required to obtain semantic training data in the first place. The authors state that "the pixel-wise local depth information can not be well represented by current depth network". However, Guizilini 2020a can generate detailed depth edges and they do NOT require any semantic information during training. The authors also state that "the required labeled semantic dataset only accounts for a very tiny proportion, which indicates a relatively lower cost." This is a bit misleading. The proposed method uses per pixel semantic ground truth from three datasets (Mapillary Vistas, Cityscapes, and KITTI). It takes a lot of effort to provide this ground truth compared to self-supervised methods which do not require any ground truth depth. It is encouraging that dataset specific semantic finetuning does not seem to have a large impact (Table 3), but this still requires access to a large enough initial semantic training set. Finally, the quantitative results are not much better than methods that don't require any semantics e.g. Guizilini 2020a, Johnston and Carneiro. Clearly, methods that do not require semantics are much more scalable, especially when adapting to new types of scenes. 


Regarding the specific contributions of the paper, the SEEM module is the most novel component of the model. However, the addition of the SEEM module does not improve quantitative performance by much (see Table 2). In addition, the qualitative improvement it provides is also very subtle. This can be seen by comparing the last two rows of Fig 7 i.e. without and with. The authors need to make a stronger case, either quantitatively or qualitatively, as to why this is valuable. 


Finally, but only a minor concern, the following relevant reference is missing: Jiao et al. Look Deeper into Depth: Monocular Depth Estimation with Semantic Booster and Attention-Driven Loss, ECCV 2018


In conclusion, there were mixed views from the reviewers - with some supportive of the paper (R2&3) others not as enthusiastic (R1&4). The authors should be commended for their detailed responses and changes already made based on reviewer comments and suggestions. Unfortunately, this did not change the mind of the reviewers. It is the opinion of this AC that there is still more work required to fully show the utility of the proposed approach, especially considering the non trivial effort that is required to obtain semantic supervision in novel domains. 

Decision: Reject