Semantic-Guided Representation Enhancement for Self-supervised Monocular Trained Depth Estimation
Authors: Rui Li
Qing Mao
Pei Wang
Xiantuo He
Yu Zhu
Jinqiu Sun
Yanning Zhang
Venue:  (2021)
Keywords: Self-supervised depth estimation
semantic-guided depth
multitask learning
semantic-guided attention mechanism

Abstract: Self-supervised depth estimation has shown its great effectiveness in producing high quality depth maps given only image sequences as input. However, its performance usually drops when estimating on border areas or objects with thin structures due to the limited depth representation ability. In this paper, we address this problem by proposing a semantic-guided depth representation enhancement method, which promotes both local and global depth feature representations by leveraging rich contextual information. In stead of a single depth network as used in conventional paradigms, we propose an extra semantic segmentation branch to offer extra contextual features for depth estimation. Based on this framework, we enhance the local feature representation by sampling and feeding the point-based features that locate on the semantic edges to an individual Semantic-guided Edge Enhancement module (SEEM), which is specifically designed for promoting depth estimation on the challenging semantic borders. Then, we improve the global feature representation by proposing a semantic-guided multi-level attention mechanism, which enhances the semantic and depth features by exploring pixel-wise correlations in the multi-level depth decoding scheme. Extensive experiments validate the distinct superiority of our method in capturing highly accurate depth on the challenging image areas such as semantic category borders and thin objects. Both quantitative and qualitative experiments on KITTI show that our method outperforms the state-of-the-art methods.

Under review as a conference paper at ICLR 2021
SEMANTIC-GUIDED REPRESENTATION ENHANCE- MENT FOR SELF-SUPERVISED MONOCULAR TRAINED DEPTH ESTIMATION
Anonymous authors Paper under double-bling blind review
ABSTRACT
Self-supervised depth estimation has shown its great effectiveness producing high quality depth maps given only image sequences as input. However, its perfor- mance usually drops when estimating on border areas or objects with thin struc- tures due the limited depth representation ability. In this paper, address this problem by proposing a semantic-guided depth representation enhancement method, which promotes both local and global depth feature representations by leveraging rich contextual information. In stead a single depth network as used in conventional paradigms, we propose an extra semantic segmentation branch offer extra contextual features for depth estimation. Based on this framework, we enhance the local feature representation by sampling and feeding the point-basee based features that locate on the semantic edges to an individual Semantic- surieeg Edge Enhancement module (SEEM), which is specifically designed for promoting depth estimation on the challenging semantic borders. Then, we improve the global fea- ture representation by proposing a semantic-guided multi-level attention mech- anism, which enhances the semantic and depth features by exploring pixel-wise correlations in the multi-level depth decoding scheme. Extensive experiments val- idate the distinct superiority of our method in capturing highly accurate depth the challenging image areas such as semantic category borders and thin objects. Both quantitative and qualitative experiments on KITTI show that our method outperforms the state-of-the-art methods.
INTRODUCTION
Depth estimation is a long standing problem in computer vision community, which offers useful in- formation to a wide range of tasks including robotic perception, augmented reality and autonomous driving, etc. Compared with depth estimation methods which rely active vision or multi-view paradigms (Schonberger & Frahm] 2016) Li et al. 2019). estimating depth from only a single image highly ill-posed, thus brings greater challenges for higher quality results.
In recent years, monocular depth estimation has witnessed new renaissances with the advent of deep learning (He et al.) 2016 Jaderberg al. 20055) Simonyan & Zisserman (014)) By learning deep representations in a supervised manner, various of networks (Eigen & Fergus 2015 Eigen capable of producing high quality depth maps thanks the large
al. 2014 Laina al. 2016) corpus of training data. At the mean time, consider the lack of labeled training data for network training. recent advances (Zhou et al. 2017 Godard et al., 2019 2017) show that monocular depth estimation can be accomplished in self-supervised way. The network can be trained by unlabeled image sequences using two-view geometric constraints, while achieving comparable results with the supervised paradigm. The learning-bass based methods managed to handle the highly ill ill-posed monocu- lar depth estimation problem by implicitly learning the mapping between visual appearance and its corresponding depth value.
However, despite the great effectiveness for learning-basee depth estimation, these methods still struggle to conduct precise depth estimation on challenging image regions such semantic cate- gory borders or thin object areas. For example, the estimated object depth usually fails to align with the real object borders, and the depth of foreground objects which have thin structures tends to be

---

Under review as a conference paper at ICLR 2021
submerged in the background. We attribute these phenomena to the limited depth representation ability that (1) the pixel-wise local depth information can not be well represented by current depth network, especially on highly ambiguous, semantic border areas, (2) current depth representations are not capable of well describing depth foreground/hacckrroudd relationships globally. These is- sues lead to the wrong depth estimation from the true scene structures, which hinders the further applications in the real-world tasks.
In this paper, we address this problem via enhancing both local and global depth feature representa- tions for self-supervised monocular depth estimation via semantic guidance. As semantic segmenta- tion conducts explicit category-level scene understandings and produces well-aligned object bound- ary detection, we propose an extra semantic estimation branch inside the self-supervised paradigm. The semantic branch offers rich contextual features which fused with depth features during multi- scale learning. Under this framework, we propose to enhance the depth feature using semantic guidance in a local-to-global way. To improve the local depth representations on semantic category borders, inspired by the sampling and enhancing strategies used in semantic segmentation (Kirillov) it al. 2020), our first contribution is to propose a Semantic-guided Edge Enhancement Module (SEEM) which is specially designed to enhance the local point-based depth representations that lo- cate on the semantic edges. Different from the method in (Kirillov et al. 2020), we enhance the point features using multi-level of representations from different domains under self-supervised framework. To be specific, we sample a set of point positions lying on the binary semantic category borders, and extract the point-wise features of the corresponding edge positions from the encoded feature, depth decoding feature as well as the semantic decoding feature. We then merge and en- hance the point-wise features and feed them to the final depth decoding features to promote the edge-aware depth inference for self- sulf-vuuervis.on For global depth representation enhancement, our second contribution is to propose semantic-gun guitted multi-level attention module to improve the global depth representation. Different from the conventional self-attentions (Fu et al. 2019 Vaswani jet al., 2017) which are implemented as single modules on the bottleneck feature block, we propose to explore the self-attentions depth representations can be further promoted by exploring and leveraging the pixel-wise correla- tions inside of their fused features.
different level of decoding layers. In this way, both semantic and
We validate our method mainly on KITTI 2015 (Geiger et al. 2012), and the Cityscapes (Cordts et al. 2016) benchmark is also used for evaluating the generalization ability. Experiments show that the proposed method significantly improves the depth estimation on category edges and thin scene structures. Extensive quantitative and qualitative results validate the superiority of our method that it outperforms the state-of-the-arr methods for self-supervised monocular depth estimation.
2 RELATED WORK
There exist extensive researches on monocular depth estimation, including geometry- based methods (Schonberger & Frahm 2016 Engvist et al. 2011) and learning- based methods (Eigen et al. 2014]] Laina et al. (2016). In this paper, however, we concentrate only on the self-supervised depth training and semantic-guided depth estimation, which is highly related to the research focus of this paper.
Self-supervised depth estimation. Sclf-supervised methods enable the networks to learn depth representation from merely unlabeled image sequences that they reformulate the depth supervision loss into the image reconstruction loss. Godard et al.) (2017) and Garg et al. (016) first propose the self-supervised method on stereo images, then Zhou al. (207)) propose monocular trained approach using a separate motion estimation network. Based on these frameworks, a large corpus of works seek promote self-supervised learning from different aspects. For more robust self- supervision signals, Mahjourian et al. (2018) propose to use 3D information as extra supervisory signal, another kind of methods leverage additional information such as the optical flow (Ranjan et al., 2019: Waan et al., 2019b) to strengthen depth supervision via consistency constraints. In order to solve the loss deviation problems on non-rigid motion areas, selective masks are used to filter out the these areas when computing losses. Prior works generate the mask by the network itself (Zhou et al., 2017: Yang et al., 2017, Vijayanarasimhan et al.) (017). while the succeeding methods produce the mask by leveraging geometric clues (Bian et al. 2019; Wang et al. 2019a Godard et al. 2019 Klingner et al. 2020). which is proved to be more effective. There also exist other methods trying enhance the network performance with traditional SfM (Schonberger & Frahm, 2016), which offer pseudo labels for depth estimation. Guizilini et al. (2020a) propose novel network architecture improve depth estimation. In this paper, we do not consider the
2

---

Under review as a conference paper at ICLR 2021
performance improvements contributed by this novel architecture in experimental comparisons, and compare different methods with conventional backbones.
Depth estimation using semantics. Semantic information shown to provide positive effeets toward depth estimation framework in the previous works. The methods can be categorized into two groups by their way to use the semantic information. One group of the methods leverage the seman- tie labels directly to guide the depth learning. The works of Ramirez et al. (2018) and (Chen et al. (2019a) propose depth constraints by leveraging the semantie labels of the scene. [Klingner et al. (020) and Casser et al. 2019) address the non-rigid motion issues by handling the moving ob- ject areas highlighted by the semantic map. Wang al. (2020) use divide-and-cone and-conquer strategy to conduct depth estimation with different semantic categories, and Zhu et (2020)) offer a depth mor- phing paradigm with the help of semantic foreground edges. The other group of methods enhance the depth representation by feature manipulation. Chen et al. (019a) generated depth and semantic maps by a single latent representation. Ochs et al. (2019 proposed segmentation-like loss item for depth estimation. Guizilini et al. (2020b) leveraged pre-trained semantic segmentation network and further conduct feature merging by PAC-based (Su et al. 2019) module.
In this paper, propose an individual semantic network for depth estimation. But different with Guizilini et al. (2020b). our semantic branch shares the same encoder and is trained together with the depth network for better scene representation. We leverage the semantic information in both ways. The semantic edges are used to guide the edge-based point feature sampling for local depth depth representation enhancement. At the mean time, we utilize the semantic feature blocks duct multi-scale feature fusion for global depth representation enhancement. The proposed method constructs a holistic way for feature representation enhancing self-supervised depth estimation.
SELF-SUPERVISED DEPTH ESTIMATION FRAMEWORK
The proposed method builds upon the self-supervised depth estimation framework. An image triplet (I--1, It+1) is used as the input, where It acts as the target image and I,' E {1--1 are the source images. During training, the target image I, is fed into the depth network fo to get the estimated depth D. = [It),, while the adjacent image pairs (Ie, Ip) are put into the ego-motion network for the 6-DoF ego-motion estimations T Then, the synthesized target images T/- can be estimated using the source images T,-, depth D2 and the ego-motions T Tr- following the formulation from Zhou et al. (2017). The self-supervised loss can be calculated by
L(,, 4x))) = Lp (4, 4e -x) + ALo 4e)
where A is the weighting factor between the photometric loss Lp and smoothness loss L.. Following Godard et al. (2019), we implement the minimum photometric loss
a Lp (x-) min( 2h (1 SS-SI (4,, (c) + (1 - a) ||4 - le-e)l)),
where SSIM is the Structural Similarity item (Wang et al.) 2004), is the weighting factor which set to 0.85. The smoothness loss item (Godard et al. (011)) is implemented to smooth the depth map by the consistency between the image and depth gradient
(4,) = ,0, D 10.44 + 10,
(3)
where 0g Og denote gradient operation x.y-axis. In our implementation, also conduct auto- masking and multi-scale depth upsampling strategy as proposed in Godard et al. (2019) to further improve depth estimation.
4 THE PROPOSED METHOD
In this paper, we promote the self-supervised depth estimation paradigm via proposing semantic- guided representation enhancing depth network. The overview of the method is shown in Figure] [ We propose a multi-task framework which consists of the depth and semantic decoding branch with shared feature encoder. During the training stage, the semantic branch feeds contextual features to the depth branch to enhance depth comprehension. Under this multi-task framework, we propose Semantic- guided Edge Enhancement Module (SEEM) to enhance the local depth feature representa- tion semantic category edges. Meanwhile, we enhance the global depth feature representations by
3

---

Under review as a conference paper at ICLR 2021
Deceding Layer:
Predicted scmantti map
GT semantic map
Semantic branch
Semantic -aninniii-gideeddd Edge Enhancement Module (SEEM)
Txceline
Layer
SA
SA
SA
Depth branch
Encoder
Semantic guided self-attemiinn
Concatenation
convolation
PFE
culancement module
Point feature
Point sampling
Point feature substitution
Estimateed depth
Figure 1: The overview of the proposed architecture. An extra semantic branch is proposed to offer contextual information for depth representations. Then, we enhance the local depth fea- ture representation via sampling and enhancing semantic edge- based point features via the proposed Semantic guided Edge Enhancement module (SEEM). The global feature representation is promoted by the proposed semantic-guided multi-layer self-attention (SA), which explores pixels feature relations between fused depth and semantic features.
proposing the semantic-guides multi-level self-attention module, which improves both the semantic and depth feature representations by digging into the pixel-wise dependencies.
4.1 THE PROPOSED SEMANTIC SEGMENTATION BRANCH
We extend the general depth network with extra semantic segmentation branch to provide explicit contextual information for depth estimation. The semantic branch shares the same network structure and input feature blocks with the depth branch. During network training, it feeds the semantic features to the depth network in a multi-layer way, to facilitate better depth representation and used for further depth representation enhancement processing. Different from general segmentation tasks, given input image 4, it outputs the binary semantic probability map MD, which implicitly encodes distance information by assigning foreground and background areas. This setting is empirically proved offer abundant guidances for better depth estimation, while leading to less computational burden. Details can be found in Section C.4 of the Appendix. Consider we build our method on the self-supervised scenarios, we use the semantic pseudo label generated by off-the-shelf method (Zhu et al. 2019) to supervise the semantic branch. We use binary cross-entropy loss to supervise the semantic branch. The semantic loss item is L, (M, Mf), where M is the binary semantic pseudo label.
4.2 SEMANTIC-GUIDED EDGE ENHANCEMENT MODULE (SEEM)
We improve depth estimation on semantic category edges via sampling and enhancing the local point features through the proposed Semantic-guidee Edge Enhancement Module (SEEM). As shown in Figure [ SEEM takes the encoding feature blocks FE, depth decoding feature blocks FD as well as the semantic decoding feature blocks Fs as input, and output a set of point features F' E RR where N is the number of sampled points, C denotes the output feature channel which consistent with the final output depth features. We first sample the local points according to the semantic edges. Then, we extract the point features from input feature blocks via grid-sampling, and enhance these feature representations via merging and feeding them into a point feature enhancement module. The resulting representative point features are used replace the final output depth features on the corresponding positions, to further promote the depth representations
semantic category edges.
comantic-guided edge point sampling strategy. A vanilla sampling strategy is to sample all points directly semantic edges. However, it shows less robustness that (1) the semantic edge is produced by pseudo labels instead of GT labels, so may not sample the real edges, and (2) the points nearby the semantic edges also challenging areas for depth estimation, which should also be taken into consideration. To address these issues, we propose a sampling strategy which incorporates enough redundancies. We first compute the semantic edges response via convolving the binary semantic
4

---

Under review as a conference paper at ICLR 2021
Semantic-gudded Edge
Sampling
Point Feature Extraction
Reference ineut
Seminti map
Encoding
Segneratio
Eiga Exrrdic
Point Feature Enhanccasen
Edge points
Disuub poins
Kandoah points
Fital points
The semantic-guided edge enhancement module (SEEM). The proposed method first samples and extracts the semantic edge-located point which consists of the edge points, disturbed edge points and random points. Then, the points features are extracted via bilinear sampling and fed into the point feature enhancement module which includes set of 1-D convolutions, to enhance the local point depth feature representations independently.
Figure 2:
mask with the Sobel operator, and then extract the edges points set Sg R which consists of μN points that exhibit largest responses from the edge response map. Besides the points lying exactly on the edges, choose another of μNN uniformly sampled random points in small range c-. and use this random point set to disturb the point positions of Se to get a new disturbed edge IRNNx2 in order sample the real ground truth edge points and nearby points. Finally, point set SR E (I-=N)xx with (1 2 points randomly sampled the global image is selected to generalize the network's 's learning abilities. The final point set E IR Nx2 is shown in Equation 4 where U is the union operator.
point set Sp
== Sg Sp UR.
(4)
Point feature extraction and enhancement. For the input feature block among Fg. FD, Fs, they are free to contain flexible layers of features. In this paper, we select features of layer (2, 1, 0) (marked with darker boxes Figure D that they are regarded to offer abundant information for image, depth, and semantic representations. We extract the point features via the point set S
FJ P(()), (E,
In Equation 5 F RNC, {E, D. denotes the extracted point features from the three input features blocks. In each input feature block, point feature extraction across different size of feature maps is conducted by bilinear interpolation (Jaderberg et al., 2015). We then feed these point-wise feature representations from different domains (e.g., image, geometry and semantic domains) to a point feature enhancement (PFE) module, which first concatenates the input features in channel- wise manner, and then enhances point representations via a set of 1-D convolutions with kernel size of 1. The output is the point feature block F'rrt E NCC
(6) where fe denotes the feature enhancement network, the output dimension is consistent with the last output features of the depth branch. We then substitute the depth feature with enhanced point feature representations F'ret the corresponding position of S. Since the module is specially designed for enhancing point-wise depth feature representations on semantic border areas, with the help of multi-domain feature representations, it's capable of conducting better depth estimation as is shown in the experiment section.
F' Fo,F',
4.3 SEMANTIC-GUIDED MULTI-LEVEL SELF-ATTENTIO
In the scene perception task, the depth information indeed, implicitly reflecting the contextual in- formation in the scene. For instance, some semantic category areas such as cars pedestrians often act as the foreground, thus they consistently exhibit smaller depth than their surrounding pixels. In method, the semantic segmentation branch provides contextual information layer of the depth branch. Consider the inherent correlation between depth and semantic represen- tations analysed above, propose the multi-level self-attention which explores the pixel-wise
each decoding
5

---

Under review as a conference paper at ICLR 2021
correlation inside the fused depth and semantic feature, to further improve the global feature repre- sentations for better depth estimation.
Conventional self-attention mechanisms are usually implemented the bottleneck features single module. In this paper, we propose the semantic-guided multi-level attention which instead consists of set of multi-level self-attentions that locate the multi-level decoding layers. The attention is conducted on both depth and semantic features. In the depth decoding stage, suppose Fbb and F3 the i-th level of feature maps from the depth and semantic branch, we concatenate the two features and fuse them via set of convolutions to get the feature map F For given fused feature FÅ E I @CC.H.xW, the self-attention module calculates the query feature, key feature and value feature {Q;, Y, V. RCix I, via convolution operation. H, and W; are height and width of features on the i-th level, and L, = H; x W,. The wise attention on the i-th level can be calculated by
A, = soJ ttmar Y?)).
Finally, the output feature can be formulated as the weighted matrix product of the value feature and the attention, added by the residual input feature
Fout = r(A: V1) + F
(8)
where y is a learnable weight factor which is initially set to O. In our paper, we implement multi- level attentions on 3 decoding layers (layer 4, 3, 2) which has smaller feature resolutions. Detailed illustrations can be found in the Appendix.
4.4
FINAL LOSS
Based on the proposed method, the final loss can be formulated as
L Lp + Lm Les
(9)
where w is set to 0.001 following common practice (Godard et al. 2019 2017 Zhou et al. 2017
Lm denotes the semantic loss as illustrated in Section
EXPERIMENTS
In this section, we conduct extensive experiments to validate the superiority of the proposed method. We use standard KITTI benchmark (Geiger et al. 2012) train and evaluate the proposed network. The Eigen's 's split (Eigen et al. 2014) along with Zhou's ' pre-processing scheme (Zhou et al.) 2017) selected which leads to 39810 images for training. 4424 images for validation and 697 images for testing. We also use the Cityscapes dataset (Cordts et al. 2016) to evaluate the generalization ability of the proposed method. The dataset is directly tested on its test split which consists of 1525 images.
5.1 IMPLEMENTATION DETAILS
Details of model training and testing. Our method is build upon Monodepth2 (Godard et al. 2019), with ResNet-50 (He et al.] 2016) initialized by the ImageNet (Deng et al.. 2009) as the backbone. The code is implemented with Pytorch (Paszke al. 2017) and trained with 1 NVIDIA Tesla V100 GPU. The input image size is 192 x 640. The motion network and depth branch are trained with self-supervision. During the point sampling process, we N 3000 that the number is enough to cover most of the edge areas on the images, is set to 0.4 by calculating the average number of edge points from randomly selected 10000 images on KITTI, and is set to 3 empirically. The network is trained with batch size 12 for 19 epochs, with learning rate of 10-4 divided by 10 after 15 epochs. The final model selected by evaluating the models run with random seeds, using the validation set.
Details of semantic label generation. For the semantic branch, the network is trained with the pseudo label dataset generated by the off-the-shelf model (Zhu et al.) 2019) (mloU of 80% on Cityscapes validation set). In our paper, the semantic model M Mcsv K is trained on Cityscapes Cordtt et al. 2016) and Mapillary Vistas dataset (Neuhold et al. 2017). and fine-tuned with KITTI Semantic (Geiger et al. 2012) (contains 200 training images) using 10-split cross validation strategy. Compared with the large quantity of KITTI raw data used for self-supervision (more than 39,000 training items), the required labeled semantic dataset only accounts for a very tiny proportion, which indicates relatively lower cost. To further validate the generalization of our method trained with

---

Under review as a conference paper at ICLR 2021
Table 1: Quantitative results on KITTI 2013. The best results are in bold and the second best sults are underlined. "S' and 'M' refer to self-suuprvisiio methods using stereo images and monocu- lar images, respectively. 'Inst' and 'Sem' mean methods that leverage instance or semantic segmen- tation information. 'PN' and 'R50' refer to the method that uses PackNet (Guizilini et al. 2020a) and Resnet-50 as backbone, respectively. All monocular trained methods are reported without post- processing steps. The metrics marked in blue mean 'lower is better', while these in red refer to 'hihher better'. Our method outperforms the state-of-the-art most metrics by a large margin.
Method
Training
Abs Rel
Sq Rel
RMSB
RMSE
1.25
1.25*
6 1.25*
Godard a1
0.133
1.142
5.533
0.230
0.936
0.970
Fillai al
2019
0.112
0.875
4.958
0.207
0.852
0.947
0.977
Watson
2019
0.106
0.780
4.695
0.193
0.875
0,958
0,980
Godard al.
2019
MS
0.106
0.806
4.630
0.193
0.876
0.958
0,980
Zhou et ul. 2017
0.183
1,595
6.709
0.270
0.734
0.902
0.959
Mahjouria al
2018
M
0.163
1.240
6.220
0.250
0,762
0.916
0.968
Yin Shi 2018
0.155
1,296
5.857
0.233
0.793
0,931
0.973
Wang et
0.151
1.257
5,583
0.974
Ranjun et 2019
0.140
1.070
5,326
0.217
0.826
0.941
0.975
Chen et al
2019
0.135
1.070
5,250
0,210
0.841
0,948
0,980
Godard
2019
0.115
0.903
4.863
0.193
0.959
0,981
Guizilini ct al.
2020aa
0.111
0.785
4.601
0.189
0.878
0.960
0.982
Johnston & Careeiro 2020
0.106
0.861
4.699
0.185
0.889
0,962
0.982
Casser
2019
M-Inst
0.141
1.026
5.291
0.215
0,816
0,945
0.979
Chen et
2019a
M+Sem
0.118
0,905
5.096
0.211
0.977
Guizilini et al.
2020b
(PN)
M+Sem
0.102
0.698
4.381
0.178
0.896
0.964
0.984
Guizilini et
2020b
(R50)
M+Sem
0.113
0.831
4,663
0.189
0.878
0.971
0.983
Ours
M+Sem
0.105
0.764
4.550
0.181
0.965
0.983
semantic labels without KITTI groundtruth guidance, offer another semantic dataset generated by Mcsv, which is trained on Cityscapes and Mapillary Vistas dataset, without involving any im- ages from KITTL. The results shown in Section 5.4 During semantic dataset generation, the raw KITTI data is fed into the pre-trained semantic model (Zhu et al.] 2019) to get the fully segmented images. Then, we get the binary semantic labels by assigning the traffic signs, different type of vehicles, and people as foreground. Details can be found in Table 7 of the Appendix.
Input
Godard et al. 2019)
Guizilini et al. 2020al
Ours
Figure Qualitative comparison on KITTI 2015. Our method shows its distinct advantages against other methods in generating sharp depth borders well as thin structures that is well-aligned with the semantic contextual information.
5.2 KITTI RESULTS
We compare our method with state-of the-art self-supervised methods including -raniet methods, monocular-trained methods as well as other semantic-guided methods. The image res- olution of the latest methods set to 192 x 640. For fair comparison, we evaluate the latest semantic-guided method (Guizilini et al. 2020b) with its ResNet-50 backbone setting instead of its PackNet setting, to rule out the performance improvement benefited from better network archi- tecture. Quantitative results are shown in T We observe that the proposed method outperforms all the current state-of-the-a stereo trained methods under the same input settings. The qualitative results shown in Figure We can see that compared with other recent state-of-the-art methods (Godard et al. 2019 Guizilini
supevsupe methods by a substantial margin, even including the
al. 2020a), method shows its remarkable advantages that method (1) precisely recovers
7

---

Under review as a conference paper at ICLR 2021
Input
Baseline
Sematic guidance
Ours
Figure 4: Qualitative results of the proposed method against the baseline. The third column
refers to the semantic guidance, in which the black and white area denote the foreground and back- ground, and the red dots are the sampled points of the proposed SEEM module.
the depth contours of the semantic objects such as humans or traffic signs, even thin structures such as human legs traffic sign poles can also be well estimated, (2) Our method precisely distinguishes between the depth foreground and background, which lead to better estimations of the global scene structures. More results can be found in the Appendix.
Table Ablation experiments. Results of several versions of our proposed method KITTI 2015
Geiger et al. 2012 are reported. The best results are in bold and the second best are underlined.
Method
SEEM
SA
Abs Rel
Sq Rel
RMSE
RMSE
5g 1.25
1.25
fg 1.25
Baseline
x
X
0.110
0.830
4.639
0.187
0.884
0.962
0.982
Baseline+SEEM
x
0.107
0.771
4.579
0.184
0.887
0.963
0.983
Baseline+SA
x
0.106
0.794
4.578
0.183
0.890
0.964
0.983
Buseline+SEEM4SS
0.105
0.764
4,550
0.181
0.891
0.965
0,983
5.3 ABLATION STUDIES
To further validate the effectiveness of the proposed contributions, We conduct ablation studies to analyze the performance improvements from individual contributions and their collaboration toward the baseline. The quantitative results are shown in Table We observe that both SEEM module and the eemantic-guided multi-level attention show distinct improvements, and their combination consistently improve the performance toward the baseline. We also show the visual improvements of our method in Figure 4 The baseline and the final results are shown in the second and final column. Pseudo semantic guidances are shown in the third column, where white/black denote the foreground/background area, the red dots represents the sampled point positions. We can see from the figure that with the help of semantic guidances, the details of the final depth maps are signif- icantly promoted, especially on the binary semantic edges, as well as the point-sampled areas. indicates that proposed contributions successfully leverage the semantic information to recover the detailed depth information on foreground semantic categories, as well as constraining the depth boundaries be aligned with the semantic object borders.
5.4 FURTHER ANALYSIS ON SEMANTICS
Analysis on different semantic models. To evaluate the influence brought by semantic models trained in different ways, we train our model with pseudo labels generated by Masv, which does not involve KITTI fine-tuning. Results are shown in Table 3 The segmentation performances of two semantic models McSV+K and Mcsy are evaluated on KITTI Semanties (Geiger al.) 2012) (200 images), both full semantic mIoU (including 19 categories from Cityscapes labels) and binary mloU (2 categories) are reported. We find interesting phenomenon that although Mcsv-Ki performance (79,900 99990%% is obviously better than Mcsv 's (67.73%) in terms of mloUFatt- both meth- ods' performances of mloU By are very close (94.74% VS 93.54%). We attribute that false segmentations often occur between categories solely belong to the background (or the foreground). Thus, the binary mask effectively alleviate the performance declines from full seman- tic labels. This indicates general advantage which still hold for other models trained with other datasets. Visual details can be found on Figure g of the Appendix. Quantitative results of Table 3 show that model trained on Mcsv semantic labels generates comparable results toward that trained on McsV+K labels. This validates the advantage of using binary semantic map, that the network performance will not be greatly influenced by different pre-trained semantic models.
the reason
Semantic category-level performance. To further validate the advantage of the semantic guidance, show the performance improvements on each semantic category toward the baseline method
8

---

Under review as a conference paper at ICLR 2021
Table Performance of models trained on different semantic datasets. The binary segmentation IoU (mloUps ps) differs a little across different datasets, and our model achieves comparable results when trained on cross-domain generated semantic labels.
Semantic model
mloUni(%)
Rel
RMSE
RMSE
5< 1.25
8 1.25"
6 < 1.25
Mcsv
79.90
94,74
0.105
0.764
4,550
0.181
0.891
0.965
0.983
Mosv
67.73
93.54
0.105
0.768
4.571
0.182
0.890
0.964
0.983
Godard ct al.(2019)
0:35
Ours
0:15
0.05
Road
Sidwwall
Raiding
Wall
Fomer
Pole
Traffic Light
Traffic Sign
Vegetation
Torain
Sky
Poosed
Rider
Car
Tmdd
Rtn
Train
Motocyyde
Blyade
Avg
Figure 5: Semantic category-level improvement. We use the AbsRel metric for evaluation. Blue refers to our methods and red denotes the baseline method (Godard et al. 2019). Benefited from the proposed SEEM module and semantic-gun guiiie multi-level attentions, our method improvements consistently on most of semantic categories.
GGdarr et al.] 2019). The results are shown Figure 5 Consider there is no groundtruth semantic labels for KITTI raw, use the pseudo labels to find semantic categories. As shown in the figure, except for category "Motorcycle" (performances are close with very small margin of 0.00016), method improves consistently across all other categories, including foreground category "traffic sign", "person". "car" and background category "sky", "building" and "vegetation". It validates that method only learns good representations of border areas, but also benefits from the global semantic contextual information provided by semantic guidance.
5.5 CITYSCAPES RESULTS
We further validate the generalization ability of our proposed method on Cityscapes (Cordts al. 2016) with models trained only on KITTI (Geiger et al. 2012). Compared with the state-of-the-an methods, our method conducts accurate and consistent depth estimation inside object categories, and recover detailed depth information on thin structures from even distant objects. Moreover, during testing, our method consistently conducts high quality predictions using either groundtruth or pseudo semantic guidance input. Additional results can be found in Figure of of the Appendix.
Ingut
Godard al. 2019)
Guizilini al. 20200
Ours-Cityp
Ours-Cityer
Figure 6: Qualitative results Cityscapes (Cordts al. 2016). All methods trained KITTI (Gigerr et al. 2012) to evaluate the generalization abilities. Ours- Ours-Citty City, and Ours- Curs-Cityer refer to results generated using input pseudo and GT semantic label, respectively.
CONCLUSION
In this paper, we propose semantic-guided feature representation enhancement paradigm for self- supervised monocular trained depth estimation. By proposing an extra semantic branch, our method learns semantic contextual information for depth estimation. We further enhance the depth feature representations local and global manner, by the proposed semantic-guie guided edge enhancement module as well as the semantic- guidee multi-layer self-attention mechanism. Extensive experiments validate the superiority of our method. In the future research, we plan to involve explicit constraints with semantic information for better depth estimation.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jiawang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, and Ian Reid. Unsupervised scale-consistent depth and ego-motion learning from monocular video. In Advances in Neural Information P Processing Systems, pp. 35-45, 2019.
Vincent Casser, Soeren Pirk, Reza Mahjourian, and Anelia Angelova. Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 8001-8008, 2019.
Po-Yi Yi Chen, Alexander Liu, Yen-Cheng Liu, and Yu-Chiang Frank Wang. Towards scene un- derstanding: Unsupervised monocular depth estimation with semantic-aware representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2624-
2632, 2019a.
Yuhua Chen, Cordelia Schmid, and Cristian Sminchiseseu. Self-supervised learning with geometric constraints in monocular video: Connecting flow, depth, and camera. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7063-7072, 2019b.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schicle. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. Proceedings of the IEEE international conference on computer vision, pp. 2650-2658, 2015.
David Figen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using multi-scale deep network. In Advances neural information processing systems, pp. 2366-2374, 2014.
Olof Enqvist, Fredrik Kahl, and Carl Olsson. Non-sequential structure from motion. In IEEE International Conference on Computer Vision Workshops, ICCV 2011 Workshops, Barcelona, Spain, November 6-13, 201 1, 2011.
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3146-3154, 2019.
Ravi Garg, Vijay Kumar Bg, Gustavo Carneiro, and Ian Reid. Unsupervised cnn for single view depth estimation: Geometry to the rescue. European conference on computer vision, pp. 740- 756. Springer, 2016.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354-3361. JEEE, 2012.
Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estima- tion with left-right consistency. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 270-279, 2017. 2017.
Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self- supervised monocular depth estimation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3828-3838, 2019.
Vitor Guizilini, Rares Ambrus, Sudeep Pillai. Allan Raventos, and Adrien Gaidon. 3d packing for self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020a.
10

---

Under review as a conference paper at ICLR 2021
Vitor Guizilini, Rui Hou, Jie Li, Rares Ambrus, and Adrien Gaidon. Semantically-guide represen- tation learning for self-super monocular depth. arXiv preprint arXiv:2002./2319 2020b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Max Jaderberg. Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Ad- vances in neural information processing systems, pp. 2017-2025, 2015.
Adrian Johnston and Gustavo Carneioo. Self-supervieed monocular trained depth estimation using self-attention and discrete disparity volume. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4756-4765, 2020.
Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In Proceedings of the IEEF/CVF Conference Computer Vision and Patlern Recognition, pp. 9799-9808, 2020.
Marvin Klingner, Jan-Aike Termöhlen, Jonas Mikolajezyk, and Tim Fingscheidt. Self-supervised monocular depth estimation: Solving the dynamic object problem by semantic guidance. arXiv preprint arXiv:2007.065 006966, 2020.
Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 2016 Fourth international confer- ence on 3D vision (3DV), pp. 239-248. IEEE, 2016.
Rui Li, Dong Gong, Jinqiu Sun, Yu Zhu, Ziwei Wei, and Yanning Zhang. Robust and accurate hybrid cructure-frrommood In 2019 IEEE International Conference on Image Processing (ICIP), pp. 494-498, IEEE, 2019.
Xiangtai Li, Xia Li, Li Zhang, Guangliang Cheng. Jianping Shi, Zhouchen Lin, Shaobua Tan, and Yunhai Tong. Improving semantic segmentation via decoupled body and edge supervision. arXiv 2020a.
preprint arXiv:2007.
Xiangtai Li, Ansheng You, Zhen Zhu, Houlong Zhao, Maoke Yang. Kuiyuan Yang, Shaohua Tan, and Yunhai Tong. Semantic flow for fast and accurate scene parsing. In European Conference on Computer Vision, pp. 775-793. Springer, 2020b.
Xiangtai Li, Houlong Zhao, Lei Han, Yunhai Tong, Shaohua Tan, and Kuiyuan Yang. Gated fully fu- sion for semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11411-11111 11425, 2020c.
Chenxu Luo, Zhenheng Yang. Peng Wang. Yang Wang, Wei Xu, Ram Nevatia, and Alan Yuille. Every pixel counts++: Joint learning of geometry and motion with 3d holistic understanding. arXiv preprini arXiv: 1810.06/15 2018
Reza Mahjourian, Martin Wicke, and Anelia Angelova. Unsupervised learning of depth and ego- motion from monocular video using 3d geometric constraints. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 5667-5675, 2018
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Buld, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In International Conference on Computer Vision (ICCV), 2017. URL htttts///wwww mapillary wtwp:coomaaasaeeceo saan/ddttaee//vitta
Matthias Ochs, Adrian Kretz, and Rudolf Mester. Sdnet: Semantically guided depth estimation network. In German Conference on Pattern Recognition, pp. 288-302. Springer, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Sudeep Pillai, Rarey Ambruş, and Adrien Gaidon. Superdepth: Self-super,ised, super-resolved monocular depth estimation. In 2019 International Conference on Robotics and Automation (ICRA), pp. 9250-9250 IEEE, 2019.
11

---

Under review as a conference paper at ICLR 2021
Pierluigi Zama Ramirez, Matteo Poggi, Fabio Tosi, Stefano Mattoccia, and Luigi Di Stefano. Ge- ometry meets semantics for semi-supervised monocular depth estimation. In Asian Conference Springer, 2018.
on Computer Viston, pp.
Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black. Competitive collaboration: Joint unsupervised learning of depth, camera mo- tion, optical flow and motion segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12240-12249, 2019.
Johannes L Schonberger and Jan-Michael Frahm. dinuctue-from-mmot revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4104-4113, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv::149. 2014.
Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik Learned-Miller, and Jan Kautz. Pixel- adaptive convolutional neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 11166-1117,, 2019.
Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger. Sparsity invariant cnns. In 2017 international conference on 3D Vision (3DV), pp. 11-20. IEEE, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is you need. Advances in neural information processing systems, pp. 5998-59980 2017.
Sudheendra Vijayanarasimhan, Susanna Ricco, Cordelia Schmid, Rahul Sukthankar, and Kate- rina Fragkiadaki. Sfm-net: Learning of structure and motion from video. arXiv preprint arXiv:1104.07804, 2017.
Chaoyang Wang, José Miguel Buenaposada, Rui Zhu, and Simon Lucey. Learning depth from monocular videos using direct methods. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 2022-2030, 2018
Guangming Wang, Hesheng Wang, Yiling Liu, and Weidong Chen. Unsupervised learning of monocular depth and ego-motion using multiple masks. In 2019 International Conference on Robotics and Automation (ICRA), pp. 4724-4730. IEEE, 2019a.
Lijun Wang, Jianming Zhang, Oliver Wang, Zhe Lin, and Huchuan Lu. Sdc-depth: Semantic divide- and-conquer network for monocular depth estimation. Proceedings of the IEEF/CVE Confer- ence on Computer Vision and Pattern Recognition, pp. 541-550, 2020.
Yang Wang, Peng Wang, Zhenheng Yang, Chenxu Luo, Yi Yang, and Wei Xu. Unos: Unified unsupervised optical-flow and stereo-depth estimation by watching videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8071-8081, 2019b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Jamie Watson, Michael Firman, Gabriel Brostow, and Daniyar Turmukhambetov. Self-supervised monocular depth hints. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2162-2171, 2019.
Ke Xian, Jianming Zhang, Oliver Wang, Long Mai, Zhe Lin, and Zhiguo Cao. Structure-guice guided ranking loss for single image depth prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 61 1-600, 2020.
Zhenheng Yang, Peng Wang, Wei Xu, Liang Zhao, and Ramakant Nevatia. Unsupervised learning of geometry with edge-aware depth-normal consistency. arXiv preprint 17Xiv:/711.03666 2017.
12

---

Under review as a conference paper at ICLR 2021
Zhichao Yin and Jianping Shi. Geonet: Unsupervised learning of dense depth, optical flow and cam- era pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1983-1992, 2018.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1851-1858, 2017.
Shengjie Zhu, Garrick Brazil, and Xiaoming Liu. The edge of depth: Explicit constraints between segmentation and depth. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 131 16-1322,,, 2020.
Yi Zhu, Karan Sapra, Fitsum A Reda, Kevin J Shih, Shawn Newsam, Andrew Tao, and Bryan Catanzaro. Improving semantic segmentation via video propagation and label relaxation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8856-
8865, 2019.
13

---

Under review as a conference paper at ICLR 2021
Appendix
A IMPLEMENTATION DETAILS
In this section, show more implementation details of our method, including network architectures and the parameter selection strategies.
A.1 NETWORK ARCHITECTURE
The network architectures are shown in Table we describe the parameters of each layer in the semantic branch, depth branch well as the proposed SEEM module. The architecture of single attention module is similar to the pixel-wise self-attention used in other methods Fu et al. (2019): Johnston & Carneiro 2020), thus do not illustrate it in detail.
Table 4: Network architectures. "in_chns" and "out_chns" denote the number of input and output channels. "resolution" refers the downscaling factor with regard the input image. "input" stands for the input of each layer, where "1" means NN-based upsampling. In the Depth Decoder, "Attn" denotes the semantic remantic-guided attentions and "SEEM" is the proposed SEEM module.
Semantic Decoder
layer
kernel
stride
ingshms
cst chhn
reselution
ingut
activation ELU
S.ipoone6
3
2048
256
32
ocoave
S_icome4
S_apeans4, ccome3
ELU
3_ppormy33
3
256
128
S_icome4
ELU
S.icom3
3
128, 512
128
8
S.urpons ecome
ELU
Sudiept
S,ionn33
Signoid
S.apoom22
5_icnme3
ELU
S_icom
64, 256
64
S. S_apeaaw2 ecomn
ELU
Jisp2
64
icvnv2
Signoid
uppcre
J.cenn2
ELU
Joone
S_apeoon ecomt
ELU
dispp
3
32
S izenv]
Sigmood
apeosry)
3
32
16
S.icone]
ELU
lcoowt
S S.upoonvi
iconst)
Sigmood
Depth Decuder
layer
kernel
siide
resdution
activation
Depperos
11.11
woad
3
256,
256
256
32
Dusposes,, S.upocesd
ELU
Aned
1024
TAID4, ecurr3
Drapeoms
25b
125
16
11.11
Purpoare', Suggores'
11117
D aomrd
228, 912
128
TAuRá, ooxrr2
BLU
Dufhept
Signol
Duppcoo22
awarr2
64,
D upcoen upoces2
ELI
Anu2
Dutlep
Rajoom??
Duppoooi
DJomn2
E1.77
ELU
D.oom
faooev ezcenil
Duliip!
32
Djcomv
Sigvvi
ELU
aumr
(bppposeti, Spppoes)
356.4
encomel) encomo))
SEEM
64.32.66
acoaro,
64,2216
3,ppoes22 sepooss) Suppardd
Dumomo
16
1
m1.17
D.dipo
Signoil
SEEM
Tayer
Impet
EndV
256, 64
128
cacopyl, cncomrl
destanty
33,
suuun,, saner 1.1 accoss)
upcomn2.
con 1
13R. 112,
256
ReLa
com
236
256
14.2
Relu
coari
256
come 1d.3
14

---

Under review as a conference paper at ICLR 2021
A.2 PARAMETER SELECTION FOR SEEM MODULE
The parameters used in SEEM module N, a and Inspired by previous works (Kirillov al, 2020 Xian et al. 2020), we set the ratio of edge-based points (including edge points Sg and disturb points Sp) to random points SR be 4 1 In this way, the ratio parameter ju is set 0.4, At the mean time, we randomly select over 10000 images from KITTI dataset to calculate the number of edge pixels, the mean value is 1233.01. To make the number of sampled edge points μ.N to be consistent with the real edge point number, we set nNN = 1200. Thus, the resulting number of sampled points N is set 3000. Under this setting, we find the sampled edge-based points (Sg, Sp) are able to cover most of the distinct edges, and the random points Sx are sampled uniformly on the whole image.
For the choice of parameter c, we compute the ratio of the edge points which are close to the GT border within range of [-c,c]. We use KITTI Semantic dataset (Geiger et al., 2012) for evaluation because it provides groundtruth KITTI semantic borders. As shown in Table when is set to 3, the ratio is 87. 35%. This means the latent sampling areas of Sp has more than 87% of overlappings with the real object borders. When is set to 5, the ratio is 92.20%, which has relative small improvement (4.45%) toward that e = 3. At mean time, since enlarging the range of the disturb points sampling will inevitably cut down the correlations between Sp and Sg, we select c = 3 as the final choice.
Table 5: Ratio of the edge points that lie within range [-c, cl of the GT border.
3
Ratio (%)
55,30
87.75
92.20
A.3 CHOICE OF ATTENTION LAYERS
In our paper, we set the number of attention layers considering both performance and GPU capacity. Firstly, we train model on images of size [64, 224). with attentions different decoding layers. The quantitative results are shown in Table 6 Compared to model with single attention (row 1), models with multi-level attentions show better performances. At the mean time, the performances of different multi-level attentions are close to each other (row 2, 3, 4). Consider the self-attention mechanism involves tensor multiplications which consume considerable GPU resources, we select model with attentions on layer (4, 3, 2) as the final choice.
Table 6: Quantitative results with different number of attentions. Results of several versions of proposed semantic- guided multi-level attentions on KITTI 2015 (Geiger al. 2012
Layers with attentions
Abs Rel
Sq Rel
RMSE
RMSE
S 1.25
S 1.25
8 S 1.25
0.140
1.103
5.479
0.218
0.821
0.943
0,976
4,3,2
0.136
1,070
5.441
0.214
0.830
0.945
0.977
4.3,2,1
0.136
1,099
5,440
0.214
0.832
0,945
0.977
4.3,2.1,0
0.137
1.065
5.391
0.214
0.828
0.945
0.977
B QUALITATIVE ABLATION ANALYSIS
We provide qualitative ablation analysis of our method. We show qualitative results of ablated ver- sions of our method Figure 7 Baseline with SEEM module produces sharper (green box in col- umn 1) and more accurate (green box in column 4) depth borders, while the baseline with semantic- guided attention (SA) generates globally accurate and smooth depth predictions (blue boxes). When combined together, the advantages of two contributions are both kept in the final predictions, which yields better results. For example, the sharper borders traffic sign on column and more accurate border predictions column 4 are kept in final results, while the final depth predictions of people are globally accurate, which are consistent with the results generated by conantic-guided attentions.
15

---

Under review as a conference paper at ICLR 2021
Input
B
B+SA B+SHEM
Ours
Figure 7: Qualitative ablation results. "B" denotes the baseline method. "B+SEEM" and "B+SA" two ablated versions of our methods. As highlighted by the green and blue boxes, the two proposed contributions complement each other to produce predictions with the best accuracy.
C FURTHER DISCUSSION ON SEMANTICS
In this section, we provide further discussion on the semantic guidance used in our method. We give qualitative comparisons between pseudo labels generated with and without KITTI fine-tuning in Section C.I Then, show the boreground//ackgroune categories used for generating binary mantic map in Section C.2 Further more, we provide quantitative comparisons between our method and the state-of-the-al segmentation methods in Section C.3 Finally, in Section C.4 evaluate
two versions of our method that implemented with binary and full semantic branches, respectively.
C.I QUALITATIVE COMPARISON BETWEEN SEMANTIC MODELS WITH/WITHOUT KITTI FINE-TUNING.
We compare the semantic prediction accuracy between Mcsv and Mcsv- The qualitative re- sults of full/binary semantic predictions are shown in Figure Row 3-4 are full semantic error map where the false predictions are marked with color. Row 5-6 are binary semantic error map, in which the false predictions are marked with gray. As shown in Figure g though the predictions between cross-domain trained Mcsv+K and KITTI fine-tuned Mcsv+K full semantic category, they are highly similar in terms the binary category. The season is that the binary mask effectively alleviate the performance declines from full semantic labels, illustrated in Section 5.4 For instance, though the "sidewalk" is falsely segmented as "road" by model trained on Mcs+v the third column, the erroneous area does not show in the binary error map because both "road" and "sidewalk" belong to the background category. This validates the advantage of using binary labels to alleviate the impact brought by cross doms-no issues.
obviously different in
C.2 GENERATION OF THE BINARY SEMANTIC MAP
Given the full semantic map, generate the binary map by assigning the foreground objects to 1, and set the rest to O. The detailed information is shown in Table
Table 7: Selection of foreground background areas.
Foreground categories
traffic sign, person, rider, car, truck, bus, train, motorcycle, bicycle, traffic light
Background categories
road, sidewalk, building, wall, fence, pole, vegetation, terrain, sky
C.3 PREDICTION ACCURACY OF THE SEMANTIC BRANCH
We show the prediction accuracy of our semantic branch using groundtruth labels of KITTI semantic dataset (Geiger et al. 2012) in Table Our model is trained on Mcsy-geeeraa semantic labels in
16

---

Under review as a conference paper at ICLR 2021
Input
Besv ECSV+K BESV EFul GT
Figure 8: Qualitative results of off-the-shelf semantic models. "E" denotes the error map, "CSV" and "CSV + K" refer results generated by model Mcsv and Mcsv+K. "Full" means full semantic label with 19 categories, while "Bi" stands for the binary label containing only back- ground/foreground categories.
order to rule out the influence KITTI groundtruth. As shown on the last two columns, our seman- tic branch learns well from the Mcsv-generatee pseudo semantic labels that their performances very close to each other. At the mean time, our semantic branch produces competitive results toward state-of-the-art semantic segmentation methods (Li et al. 2020b in terms of binary segmen- tation, which shows the effectiveness of our method in providing reliable foreground/backgrounn predictions.
Table 8: Binary semantic segmentation accuracy. We compare the binary prediction accuracy among our semantic branch, supervisory pseudo labels and other semantic segmentation methods.
Method
Li al.
2020b
Li al. (020cc
Li at.
2020a
Ours
Mosv
mloUp: (%)
89.23
92.43
92.94
92.59
93.54
DICE (%)
95.95
94.07
96.58
96.05
96.24
C.4 COMPARISON BETWEEN THE IMPLEMENTATION OF BINARY/FULL SEMANTIC SEGMENTATION BRANCHES
To further validate the effectiveness of the proposed binary semantic branch, we conduct quan- titative comparison between the semantic segmentation branches supervised by full/binary labels respectively. For the semantic branch trained by full semantic labels, we change the network to con- duct pixel-wise predictions of 19 categories. The full semantic labels are pre-processed into one-hot vectors for supervision. Different from the network settings illustrated in the main text, the semantic features are directly concatenated with the depth feature without additional contributions. The re- sults shown in Table validate that there is distinct performance difference between the networks supervised by full or binary semantic labels. Thus, we choose the binary semantic branch in our final settings it well encodes the semantic contextual information with the preferable simplicity.
Table 9: Quantitative comparison between different semantic segmentation branches. "Full semantic label" refers to the semantic branch supervised by full category labels (19 categories), while "Binary semantic label" represents the model trained by the binary semantic label.
Groundtruth type
Abs Rel
Sq Rel
RMSE
RMSE
< 1.25
8 < 1.25
< 1.25
Full semantic label
0.107
0.835
4.675
0.185
0.963
0.982
Binary semantic label
0.108
0.784
4.588
0.184
0.887
0.963
0.983
17

---

Under review as a conference paper at ICLR 2021
Table 10: Quantitative results on KITTI improved ground truth. "S" and "M" refer self- supervision methods trained using stereo images and monocular images, respectively. Results are presented without any post-process post-processing. The metrics marked in blue mean "lower better", while these in red refer to "higher is better". Our method still produce comparable or better results com- pared to the state-of-the-ar methods.
Method
Training
Ahs Rel
Sq Rel
RMSH
RMSF
6 < 1.25
4 H 1,25
Ó 1.25*
Godard m
0.109
0.811
4.568
0.166
0.988
Pillai al.
2019 +pp
0.090
0.542
3.967
0.144
0.901
0.976
0.993
Godard al
0.085
0.537
3.868
0.139
0.912
0.979
0.993
2018
MS
0.123
0.754
4,453
0.172
0.863
0.964
0.989
Godard ct al
2019
MS
0.080
0.466
3.681
0.127
0,926
0.985
0.995
Zhou al.
2017
0.176
1.532
6.129
0.244
0.921
0.971
Mahjourian al.
2018
M
0.134
0.983
5.501
0.203
0.827
0,944
0.981
Shi
2018
0.132
0,994
5.240
0.193
0.953
0.985
Wang et 2018
0.126
0.866
4,932
0.185
0.851
0.958
0.986
Ranjan et
2019
0.123
0.881
4,834
0.181
0.860
0.959
0.985
EPC++
0.120
0.789
4.755
0.177
0.856
0.961
0.987
Godard et ul.
2019
0.090
0,545
3,942
0.137
0.914
0.983
0.995
Guzzilini
20204
0.078
0,420
3,485
0.121
0.931
0.986
0.996
Ours
0.081
0.437
3.635
0.124
0.927
0.987
0.996
D KITTI IMPROVED GROUND TRUTH RESULTS
We KITTI (Geiger et al. 2012) with improved groundtruth (Uhrig et al. 2017 to further validate the effectiveness our method. The test set is part of the Eigen split (Eigen et al.) 2014), which accounts for 93% (652 / 691) of the original test frames. Quantitative results shown in Table 10 Our method produces comparable or better results toward other state-of-the-art methods.
E ADDITIONAL QUALITATIVE RESULTS
We offer additional qualitative comparisons for depth on benchmark KITTI Geiger et al. 2012) and Cityscapes (Cordts al. 2016). All models are trained using only KITTL As shown in Figure 9 (KITTI) and Figure 10 (Cityscapes), the proposed method achieves the best performance, that produces sharp depth edges well detailed depth information from thin structures. Note that in Figure [0] the fourth and fifth column indicate depth maps generated with pseudo semantic guid- ance and groundtruth guidance, respectively. Results show that even tested with pseudo semantic guidance, our method still yields high quality depth predictions as the one guided with groundtruth.
18

---

Under review as a conference paper at ICLR 2021
Input
Godard et al. 2019
Guizilini et al. (200aa)
Ours
Figure 9: Additional qualitative comparisons on KITTI dataset. All models are trained on KITTI
(Geiger al. 2012) Eigen split (Bigen al. 2014).
19

---

Under review as a conference paper at ICLR 2021
Input
et
Ours-Cityp
Ours-Cityer
Figure 10: Additional qualitative comparisons Cityscapes. All models are trained KITTI (Geiger et al. 2012) Eigen split (Eigen et al. 2014). Ours- City, and Ours-Citycr refer to results generated using input pseudo and GT semantic label, respectively.
20


Reviews:
[{"review_id": "qGbhXLMUkw", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n1. Writing is good. It is easy to read.\n2. Although limited, the proposed term really improves the performance.\n3. The qualitative results show better depth estimation in object boundary areas. \n\n\n\nWeakness:\n\n4. “How to generate semantic labels” is a very important problem in this paper. Simply referring to (Zhu et al., 2019) is not sufficient, and I suggest authors at least using 3-5 sentences to describe that, covering introduction for the input, output, and methodology.\n\n5. After reading (Zhu et al., 2019), I find that the ground-truth semantic labels are still required, and their method can be regarded as a data augmentation. Then I have serious concerns about semantic supervisions:\na. The ground truth for semantics requires human labelling, while the depth supervision needs LiDAR or Kinect-like sensors. In my opinion, the former one is more expensive than the latter, so the requirement of training the proposed model is higher than existing supervised methods.\nb. By comparing the results, I find that the proposed method is not as good as supervised methods or self-supervised methods that use stereo pairs for training. \nc. Due to the requirement for ground truth semantics, the claim of self-supervision is not convincing.\nd. It loses the advantage of self-supervised depth estimation. For example, existing depth CNN (Gordon et, al, iccv 2019) can be trained on a wide range of YouTube videos, where no semantic labels are provided. Although the pre-trained segmentation network could be used, their generalization is not validated.\n\n6. The proposed method leads to very limited performance improvement, e.g., final results (absrel=0.105) vs baseline results (absrel=0.110) as shown in the Table 2, while it requires additional semantic labels for training. Considering the trade-off between performance improvement and requirement for training data, I have serious concerns about the contribution of this paper. \n\n7. Object boundary is hard to estimate in all dense prediction tasks. This paper improves that in depth estimation by using the well-labeled semantic maps. However, in practice, obtaining accurate semantic labels is difficult, and this is also a challlenging problem for semantic segmentation. \n\n8. The importance of depth sharpness on object boundary should be more clearly discussed, and the trade-off between such improvement and disadvantages caused by proposed solutions (as I mentioned above) should be discussed.\n\npost-rebuttal: Authors pay much effort on addressing issues that I mentioned, and I appreciate that very much. My issues can be partially resoved, but the main issue is regarding the high-level thinking on semantic supervisions, which is impossible to be fully addressed in a rebuttal. Besides, I agree with AC that (a) the improvement is very limited and (b) semantic labels are hard to obtain. Overall, I raise my rating to 5 (not so good, but could be accepted), because I really appreciate authors' effort.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "c4jUejeK9qx", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method for semantic-guided self-supervised depth estimation from monocular images. They propose semantic guidance to improve depth estimation performance. This is obtained by applying semantic guidance at multiple levels in the decoder via an attention layer and via feature enhancement in edges areas of the image. Experiments show the proposed method reaches state of the art performance for self-supervised monocular depth estimation in structure-from-motion setting.\n\nPros\n- given semantic guidance, this method seems well designed and effective to enhance depth estimation performance with a relatively low computation overhead in order to provide semantic guidance\n- interesting semantic guided edge point sampling strategy\n- the paper shows an improved semantic guidance with respect to Guinzilini et al 2020b\n\nCons\n- not clear how the decoder was designed, why on some layer there is attention and on other concatenation + convolution? Is it experimentally obtained, a design choice? Could the authors give some explanation about it?\n- personally I find difficult to understand the choice of parameters \\mu, c and N related to the semantic guided edge point sampling strategy, could the authors provide some detail about it?\n- about the point feature extraction and enhancement, two unclear parts for me. First, \"we select the 0 − 2 layers of features\" could you explain what features/feature blocks you are referring to? Maybe a small explanation could be useful or adding in Fig. 1 some corresponding number to relate to. Second, \"point feature enhancement module, which merges and enhances these point representations via a set of 1-D convolutions\", the PFE is not clear, set of 1-D convolution means that for each channel of the feature map there is a convolution?\n- Eq. 8, i \\in [2,4] is related to the darker box around the decoder blocks in Fig. 1?\n- the training set for Cityscape is the train split with fine annotations or also with coarse annotations?\n\nOverall, the paper is well written, the results are sound and minor improvements could make it worth acceptance.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VXvx5YpGI2Y", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors tackle the problem of self-supervised depth estimation and particularly address the issue of poor depth estimation on object boundaries. The introduction motivates the problem well, and the related work covers most of the relevant papers. The authors propose two main modifications allowing them to leverage an off-the-shelf semantic segmentation network: SEEM (semantic guided edge enhancement module) and a multi-level self-attention mechanism that fuses depth and semantic features at different levels. The combination of these contributions along with an off-the-shelf semantic segmentation network achieves impressive results, especially on the  boundaries of objects. \n\nOverall the paper is well written and the method clearly explained; the 2 diagrams in Figures 1 and 2 are very well done and explain the workings of the system quite well. The authors ablate their contributions clearly showing their respective impact on the overall performance. The evaluation on KITTI shows that the method compares favorably with related work. The presentation could be improved by addressing the points mentioned below, and specifically: (i) it would be interesting to see how the quality of the off-the-shelf semantic segmentation network affects the depth results (see below) and (ii) it would be nice to see a per semantic class evaluation (similar to Fig 4 in  (Guizilini et al., 2020b)) showing specific improvements after using the proposed modifications. \n\nComments/suggestions for improvement:\n\n* Does the quality of the off-the-shelf segmentation affect the result? There is an implicit assumption here that the pretrained, off-the-shelf network transfers well to the current domain. It would be interesting to explore how this affects depth estimation performance. \n* What data is the pretrained semantic segmentation network of Zhu et al. (2019) trained on? Could the authors report the performance of the network on the original training dataset, for reference? Guizilini et al. (2020b) report that their pre-trained semantic segmentation network achieves an mIoU of 75% on the Cityscapes val set; how does that compare to the network of Zhu et al. (2019) used by the authors?\n* Table 1 - the numbers reported for Guizilini et al. (2020b) don’t seem to include the PackNet numbers reported in that paper (i.e. the abs_rel reported for 640x192 seems to be 0.102); while that network architecture is different, I would argue it’s still relevant as a comparison with state-of-the-art methods\n* Table 3 in the supplementary is missing the Guizilini et al. (2020a) numbers on the KITTI improved ground truth.\n* The paper is missing some details regarding the network architectures used - the authors should provide this information if possible, either in the main text or in the supplementary\n* It would also be interesting to see the performance of the method with using full resolution images - would using semantic segmentation further help improve results in that setting?\n\nMinor comments:\n* Table 1 caption “D” denotes depth supervision - since there is no method using Depth supervision in the table this can be removed\n* It would be nice if the authors could add F_E, F_D and F_S on Fig 1 to make it easier to follow the description in Sec 3.2\n* Setting i \\in [2,4] means using the self attention twice? Fig 1 has 3 self-attention blocks.\n\nPost rebuttal: I thank the authors for their detailed response and paper revision. My concerns have been addressed, and I particularly appreciate the experiments presented Table 3, Figure 5 and Section 5.4. I am happy to maintain my rating and recommend acceptance. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "AGlWaiBd2su", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposed a novel framework to improve self-supervised monocular depth estimation leveraging semantic features at local and global level. The proposed framework includes a semantic-guided edge enhancement module to extract and enhance point-based features around semantic boundaries. The proposed framework also incorporates feature fusion through a self-attention module at different feature levels for global fusion. Evaluation on KITTI datasets compared with recent state-of-the-arts are provided. The proposed method outperformed other monocular depth estimation methods including the ones that also leverage semantic information.\n\n--------------------------------------------------------------------------------------------\n\nPros: \n* The paper is well formatted and easy to read. \n* The idea of semantic guidance both locally and globally is novel and interesting.\n* The new framework performs the previous state-of-the-arts under the same network settings. \n\nCons:\n* The cityscapes experiment provides very limited contribution to the paper as only qualitative results are provided.\n* The prediction accuracy of semantic segmentation branches is missing. Even Though it is trained with pseudo labels, it is still important to provide this result in order to justify that the semantic branch is providing reasonable predictions. \n* There is some other confusing description in the experiment section that raises concerns:\n  - The paper keeps mentioning that the semantic pseudo labels are generated from off-the-shield algorithms which require no groundtruth. This argument is confusing as the pretrained model from reference work did use ground truth. \n  -The authors mentioned semantic labels for the test set of KITTI and Cityscapes. Why do you need to use these labels for testing when no semantic quantitative result is provided? Please provide more details there. Or one might assume the label is needed during network inference. \n  - The author mentioned that the final model is selected via validation set. Does that mean the reported number is not from a model converged with the training protocol? Or is it selected from multiple runs with random seed? Please provide more details there. \n\n----------------------------------------------------------\nOther detail comments:\n\n- In related work, self-supervised depth estimation, I think it would be good to mention the effort of network architecture improvement for depth estimation (E.g. Guizilini 2019a). Then emphasize that the experiment comparison rules out the impact of different network architecture in either related work or experiment section, given that the main result in Guizelini 2019b using another network is not compared in the table.\n- Since both depth branches and semantic branches are trained together, how do you weigh the losses? It is a simple sum up or weighted? Please provide the detail for reproducibility.\n- In figure 4. Is the semantic guidance map the groundtruth label or predicted maps?\n- The author mentioned in Appendix C that binary semantic categorization is used. Please provide details on what classes are considered as foreground. \n- The description “insufficient depth representation” is a bit confusing. The solution to “insufficient” should be “providing more”, while the paper is improving and augmenting the representation. I would recommend the author consider changing the description here. \n- It would be nice to include some qualitative results along with the ablation analysis. \n--------------------------------------------------------\npost-rebuttal:\nI thank the authors for their clarification and ablation analysis. Most of my concerns are resolved. I will keep my original score and I think the paper could be accepted. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors address the problem of self-supervised monocular depth estimation via training with only monocular videos. They propose to use additional information extracted from semantic segmentation at training time to (i) provide additional “semantic context” supervision and (ii) to improve depth estimation at discontinuities through an edge guided point sampling based approach. Results are presented on the KITTI and Cityscapes datasets.  

One of the main concerns is related to the utility of the semantic supervision given the relative cost required to obtain semantic training data in the first place. The authors state that "the pixel-wise local depth information can not be well represented by current depth network". However, Guizilini 2020a can generate detailed depth edges and they do NOT require any semantic information during training. The authors also state that "the required labeled semantic dataset only accounts for a very tiny proportion, which indicates a relatively lower cost." This is a bit misleading. The proposed method uses per pixel semantic ground truth from three datasets (Mapillary Vistas, Cityscapes, and KITTI). It takes a lot of effort to provide this ground truth compared to self-supervised methods which do not require any ground truth depth. It is encouraging that dataset specific semantic finetuning does not seem to have a large impact (Table 3), but this still requires access to a large enough initial semantic training set. Finally, the quantitative results are not much better than methods that don't require any semantics e.g. Guizilini 2020a, Johnston and Carneiro. Clearly, methods that do not require semantics are much more scalable, especially when adapting to new types of scenes. 


Regarding the specific contributions of the paper, the SEEM module is the most novel component of the model. However, the addition of the SEEM module does not improve quantitative performance by much (see Table 2). In addition, the qualitative improvement it provides is also very subtle. This can be seen by comparing the last two rows of Fig 7 i.e. without and with. The authors need to make a stronger case, either quantitatively or qualitatively, as to why this is valuable. 


Finally, but only a minor concern, the following relevant reference is missing: Jiao et al. Look Deeper into Depth: Monocular Depth Estimation with Semantic Booster and Attention-Driven Loss, ECCV 2018


In conclusion, there were mixed views from the reviewers - with some supportive of the paper (R2&3) others not as enthusiastic (R1&4). The authors should be commended for their detailed responses and changes already made based on reviewer comments and suggestions. Unfortunately, this did not change the mind of the reviewers. It is the opinion of this AC that there is still more work required to fully show the utility of the proposed approach, especially considering the non trivial effort that is required to obtain semantic supervision in novel domains. 

Decision: Reject