Semantically-Guided Representation Learning for Self-Supervised Monocular Depth
Authors: Vitor Guizilini
Rui Hou
Jie Li
Rares Ambrus
Adrien Gaidon
Venue:  (2020)
Keywords: computer vision
machine learning
deep learning
monocular depth estimation
self-supervised learning

Abstract: Self-supervised learning is showing great promise for monocular depth estimation, using geometry as the only source of supervision. Depth networks are indeed capable of learning representations that relate visual appearance to 3D properties by implicitly leveraging category-level patterns. In this work we investigate how to leverage more directly this semantic structure to guide geometric representation learning, while remaining in the self-supervised regime. Instead of using semantic labels and proxy losses in a multi-task approach, we propose a new architecture leveraging fixed pretrained semantic segmentation networks to guide self-supervised representation learning via pixel-adaptive convolutions. Furthermore, we propose a two-stage training process to overcome a common semantic bias on dynamic objects via resampling. Our method improves upon the state of the art for self-supervised monocular depth prediction over all pixels, fine-grained details, and per semantic categories.


Published as a conference paper at ICLR 2020
SEMANTICALLY-GUDDED REPRESENTATION LEARN- ING FOR SELF-SUPERVISER MONOCULAR DEPTH
Vitor Guizilini!
Rui Hou
Jie Li
Rareş Ambrus'
Adrien Gaidon' University of Michigan rayhou8 umi ch edu
Toyota Research Institute (TRI) {first. Last pnftri.giobd
ABSTRACT
Self-supervised learning is showing great promise for monocular depth estima- tion, using geometry the only source of supervision. Depth networks indeed capable of learning representations that relate visual appearance to 3D properties by implicitly leveraging category-level patterns. In this work we investigate how to leverage more directly this semantic structure to guide geometric representa- tion learning, while remaining in the self-supervised regime, Instead of using semantic labels and proxy losses in a multi-task approach, we propose new ar- chitecture leveraging fixed pretrained semantic segmentation networks to guide self- self-supervised representation learning via pixel-adaptive convolutions. Further- more, we propose a two-stage training process to overcome common semantic bias on dynamic objects via resampling. Our method improves upon the state of the art for self--upervised monocular depth prediction over all pixels, fine-grained details, and per semantic categories.
INTRODUCTION
Accurate depth estimation is a key problem in computer vision and robotics, as it is instrumental for perception, navigation, and planning. Although perceiving depth typically requires dedicated sensors (e.g., stereo rigs, LiDAR), learning to predict depth from monocular imagery can provide useful cues for a wide array of tasks (Michels et al., 2005; Kendall et al., 2018; Manhardt et al., 2019; Lee et al., 2019). Going beyond supervised learning from direct measurements (Eigen et al., 2014), self-supervised methods exploit geometry as supervision (Guo et al., 2018; 88; Pillai et al., 2019; Zou et al., 2018; Yang et al., 2017), therefore having the potential to leverage large scale datasets of raw videos to outperform supervised methods (Guizilini et al., 2019).
Although depth from single image is ill-posed inverse problem, monocular depth networks able to make accurate predictions by learning representations connecting the appearance of scenes and objects with their geometry in Euclidean 3D space. Due perspective, there is indeed an equivariance relationship between the visual appearance of an object in 2D and its depth, when con- ditioned on the object's 's category. For instance, car 25 meters away appears smaller (on the image plane) than a car only 5 meters away but bigger than truck 50 meters away. Current depth estima- tion methods either do not leverage this structure explicitly or rely on strong semantic supervision to jointly optimize geometric consistency and semantic proxy task in a multi-task objective (Ochs et al., 2019; Chen et al., 2019), thus departing from the self-supervised paradigm.
In this paper, we explore how we can leverage semantic information to improve monocular depth prediction self-supervised way. Our main contribution is novel architecture that uses fixed pre-trained semantic segmentation network to guide geometric representation learning in self- supervised monocular depth network. In contrast to standard convolutional layers, our architecture uses pixel-adaptive convolutions (Su et al., 2019) to learn semantic-depen dependent representations that can better capture the aforementioned equivariance property. Leveraging semantics may nonetheless introduce category-specifie biases. Our second contribution is two-stage training process where we automatically detect the presence of a common bias on dynamic objects (projections at infinity) and resample the training set de-bias it. Our method improves upon the state of the in self- supervised monocular depth estimation on the standard KITTI benchmark (Geiger et al., 2013), both on average over pixels, over classes, and for dynamic categories in particular.
Source code and pretrained models are available on https://githuh.com/TT MRI--//-cc/////MMIIIIII http://////R.Ro.mmmpaaeetttiitii

---

Published as a conference paper at ICLR 2020
Example of a pointcloud generated using our proposed semantically-guiie architec- ture, colored by RGB values from the input image and corresponding predicted semantic labels.
Figure 1:
2 RELATED WORK
Since the seminal work of Eigen et al. (2014), substantial progress has been done to improve the accuracy of supervised depth estimation from monocular images, including the of Conditional Random Fields (CRFs) (Li et al., 2015), joint optimization of surface normals (Qi et al., 2018), fusion of multiple depth maps (Lee et al., 2018), and ordinal classification (Fu et al., 2018). Con- sequently, as supervised techniques for depth estimation advanced rapidly, the availability of large- scale depth labels became a bottleneck, especially for outdoor applications. Garg et al. (2016) and Godard et al. (2017) provided an alternative self-supervised strategy involving stereo cameras, where Spatial Transformer Networks (Jaderberg et al., 2015) can used to geometrically warp, in a dif- ferentiable way, the right image into a synthesized left image, using the predicted depth from the left image. The photometric consistency loss between the resulting synthesized and original left images can then be minimized in end-to-end manner using Structural Similarity term (Wang et al., 2004) and additional depth regularization terms. Following Godard et al. (2017) and Um- menhofer et al. (2017), Zhou et al. (2017) generalized this to the purely monocular setting, where a depth and a pose networks are simultaneously learned from unlabeled monocular videos. Rapid progress in terms of architectures and objective functions (Yin & Shi, 2018; Mahjourian al., 2018; Casser et al., 2019; Zou et al., 2018; Klodt & Vedaldi, 2018; Wang et al., 2018; Yang et al., 2018) have since then turned monocular depth estimation into one of the most successful applications of self-supervised learning, even outperforming supervised methods (Guizilini et al., 2019).
The introduction of semantic information to improve depth estimates has been explored in prior works, and can be broadly divided into two categories. The first one uses semantic (or instance) in- formation to mask out or properly model dynamic portions of the image, which are not accounted for in the photometric loss calculation. Güine& & Geiger (2015) leveraged object knowledge in a Markov Random Field (MRF) to resolve stereo ambiguities, while Bai et al. (2016) used a conjunction of instance-leved segmentation and epipolar constraints to reduce uncertainty in optical flow estimation. Casser et al. (2019) used instance-level masks to estimate motion models for different objects in the environment, and account for their external motion in the resulting warped image. The second cat- egory attempts to learn both tasks in a single framework, and uses consistency losses to ensure that both are optimized simultancously and regularize each other, so the information contained in one task can be transferred to improve the other. For instance, Ochs et al. (2019) estimated depth with an ordinal classification loss similar to the standard semantic classification loss, and used empirical weighting to combine them into single loss for optimization. Similarly, Chen et al. (2019) used a unified conditional decoder that can generate either semantic or depth estimates, and both outputs are used to generate series of losses also combined using empirical weighting to generate the final loss to be optimized.
Our approach focuses instead on representation learning, exploiting semantic features into the self- supervised depth network by using a pretrained semantic segmentation network to guide the gen- eration of depth features. This is done using pixel-adaptive convolutions, recently proposed in Su et al. (2019) and applied to tasks such as depth upsampling using RGB images for feature guidance. We show that different depth networks can be readily modified to leverage this semantic feature guidance, ranging from widely used ResNets (He et al., 2016) to the current sfate-of -fate-of-the-rt PackNer (Guizilini et al., 2019), with consistent gain in performance across these architectures.
2

---

Published as a conference paper at ICLR 2020
3 SELF-SUPERVISED STRUCTURE-
Our semantically-guidee architecture is developed within a self-supervised monocular depth esti- mation setting, commonly known structure-from-mottiin (SIM). Learning in a self-supervised structure- structure-from-mmot from-motion setting requires two networks: a monocular depth model fo + D, that outputs a depth prediction D = (I(( for every pixel p in the target image I; and a monocular ego-motion estimator fx (Te, Is) + Xt-5, that predicts the 6 DoF transformations for all E S given by Xt-2s = ( } E SH(3) between the target image I, and a set of temporal context source images 11 E Ig. all reported experiments we use 4--1 and 441 as source images.
3.1 THE SELF-SUPERVISEE OBJECTIVE LOss
We train the depth and pose networks simultaneously, using the same protocols and losses as de- scribed in Guizilini et al. (2019). Our self-supervised objective loss consists of an appearance match- ing term Lp that is imposed between the synthesized I and original I target images, and depth regularization term L, that ensures edge-aware smoothing in the depth estimates δt. The final objective loss is averaged per pixel, pyramid scale and image batch, and is defined as:
[(,, f.) = (T, f.) A (D:))
where A is weighting coefficient between the photometric L. and depth smoothness L., loss terms. Following Godard al. (2017) and Zhou et al. (2017), the similarity between synthesized i, and original I, target images is estimated using a Structural Similarity (SSIM) term (Wang et al., 2004) combined with an L1 loss term, inducing the following overall photometrie loss:
(T, f.) a SSIM 1t) + - a) 1 f
2
In order to regularize the depth in low gradient regions, we incorporate an edge- term similar Godard al. (2017). This loss is weighted for each of the pyramid levels, decaying by a factor of 2 each downsampling. starting with weight of 1 for the 0 pyramid level.
(D.) = 1o_ D.le-
18, 1.le__111111 16,44l
We also incorporate some of the insights introduced in Godard et al. (2018), namely auto-masking. minimum reprojection error, and inverse depth map upsampling to further improve depth estimation performance in our self-supervised monocular setting.
3.2 DEPTH AND POSE NETWORKS
Our baseline depth and pose networks are based on the PackNer architecture introduced by Guizilini et al. (2019), which proposes novel packing and unpacking blocks to respectively downsample and upsample feature maps during the encoding and decoding stages. This network was selected due to its tatteoffthhe-a performance in the task of self-supervised monocular depth estimation, can analyze our proposed architecture is capable of further improving the current state-of-the-ard which models our proposed semantically- -nuuidd architecture
However, there no restrictions can be applied to, and in Section 5.4 we study its application to different depth networks.
4 SEAANTICALLY-UIDEEE GEOMETRIC REPRESENTATION LEARNING
In this section, describe our method to inject semantic information into self-supervised depth network via its augmentation with semantic-aware convolutions. Our proposed architecture is de- picted in Figure 2 and is composed of two networks: primary one, responsible for the generation of depth predictions Đ- = (T())): and a secondary one, capable of producing semantic predic- tions. Only the first network is optimized during self-supervised learning: the semantic network is initialized from pretrained weights and is not further optimized. This is in contrast to the common practice of supervised (ImageNet) pretraining of depth encoders (Godard et al., 2018; Casser et al., 2019; Zou et al., 2018): here instead of fine-tuning from pre-trained weights, we preserve these sec- ondary weights guide the feature learning process of the primary depth network. Our approach also differs from learning without forgetting (Li & Hoiem, 2017) by leveraging fixed intermediate feature representations way maintain consistent semantic guidance throughout training.
3

---

Published as a conference paper at ICLR 2020
Pretrained Semartic Network
Featura
Fived
Guidance
Guidance
Conv.
Feature
Geoomtiic:
Feature
Bemantic Guidance
Depth Network
Figure 2: Diagram of our proposed architecture for self-supervised monocular depth estimation with semantically-guide feature learning. The semantic network is fixed and initialized from pre- trained weights, while the depth network is trained end-to-end in a self-supervised way, including pixel-adaptive convolutions (Guidance) on its decoder to learn semantic-dependen geometric fea- tures.
4.1 SEMANTICALLY-GUIDT DEPTH FEATURES
We leverage the information from the pretrained semantic network in the depth network through the use of pixel-adaptive convolutions (Su et al., 2019). They were recently proposed to address some limitations inherent to the standard convolution operation, namely its translation invariance making it aantent-gnnoticc While this significantly reduces the number parameters of the resulting net- work, this might also lead to sub-optimal solutions under certain conditions important for geometric representation learning. For example, spatially-shared filters globally average the loss gradients over the entire image, forcing the network to learn weights that cannot leverage decation-speciinn information beyond their limited receptive fields. Content-agnostic filters are unable to distinguish between different pixels that are visually similar (i.e. dark areas due shadows or black objects) or generalize to similar objects that are visually different (i.e. cars with varying colors). In this work, we use tixeladdptivve convolutions produce semantic-aware depth features, where the fixed in- formation encoded in the semantic network is used to disambiguate geometric representations for the generation of multi-level depth features.
As shown in Figure 2, we extract multi-level feature maps from the semantic network. For each feature map, we apply 3 x and a (Wu & He, 2018) and ELU non- non-lineasisies (Clevert et al., 2016). These processed semantic feature maps are then used as guidance on their respective pixel-adaptive convolutional layers, following the formulation proposed in Su et al. (2019):
convolutional layer followed by Group Normalization
Σ W(f. p,ly, +b JEA(()
(4)
In the above equation, f' RD are processed features from the semantic network that will serve to guide the pixel-adaptive convolutions from the depth network, p (x, y)² are pixel coordinates, with [p: pj] denoting 2D spatial offsets between pixels, W Waxk are convolutional weights with kernel size k. Ω. defines a k x i convolutional window around i, v is the input signal to be convolved, and Rl is a bias term. K is the kernel used calculate the correlation between guiding features, here chosen to be the standard Gaussian kernel:
K (f., = exp (- -f, (f. 12)
=xxp
(5)
where Egy is the covariance matrix between features f, and fy, here chosen to be a diagonal matrix o². Tp, with a as an extra learnable parameter for each convolutional filter. These kernel evaluations can be seen as a secondary set of weights applied to the standard convolutional weights, changing their impact on the resulting depth features depending on the content stored the guiding semantic features. For example, the information contained in depth features pertaining not be used to generate depth features describing a pedestrian, and this behavior is now captured as larger distance between their corresponding semantic features, which in turn produces smaller weights for that particular convolutional filter. Note that the standard convolution can be considered special case of the pixel-adaptive convolution, where V ij, K (f,,,f,) 1.
the sky should
4

---

Published as a conference paper at ICLR 2020
Figure 3: Qualitative results our proposed two-stage training to address the infinite depth problem. Top images were obtained evaluating the first-stage depth network, and bottom images were obtained using the second-stage depth network, trained with a filtered dataset.
4.2 SEMANTIC GUIDANCE NETWORK
As the secondary network used to provide semantic guidance for the generation of depth features, we use Feature Pyramid Network (FPN) with ResNet backbone (Lin et al., 2017). This architecture has been shown to be efficient for both semantic and instance-level predictions towards panoptic segmentation (Kirillov et al., 2019; Li et al., 2018; Xiong et al., 2019; Porzi et al., 2019). While our proposed cemantically-guide this particular implementation to facilitate the future exploration of different sources for guidance information. Architectural details follow the protocols described in Li et al. (2018), and unless mentioned otherwise the same pretrained model was used in all reported experiments. The semantic network is assumed fixed, pretrained on held out dataset different than the raw data used for self- supervised learning, i.e. we do not require any semantic ground truth on the target dataset.
architecture is not restricted to any particular network, we chose
4.3 TTOGS TRAINING
One well-known limitation of the self-supervised photometric loss is its inability model dynamic objects, due to a static world assumption that only accounts for camera ego-motion (Godard et al., 2018: Casser al., 2019). A resulting common fallure mode is the infinite depth problem, which is caused by the presence of objects moving at the same speed as the camera. This typically causes distinct holes in the predicted depth maps, with arbitrarily large values where these objects should be. This severely hinders the applicability of such models in real-world applications, particularly for automated driving, where the ability to detect and properly model dynamic objects is crucial. More- over, this limitation may be further accentuated in our proposed semantically-gui -uided architecture, as the infinite depth problem occurs mostly on dynamic categories (i.e. cars and motorcycles) and the semantic- features may reinforce this bias.
We propose a simple and efficient two-stage training method to detect and remove this bias from the training set. In the first stage, we learn standard depth network all available training data. This network, exhibiting the infinite depth problem, is then used to resample the dataset by automatically filtering out sequences with infinite depth predictions that violate basic geometric prior. We indeed find that depth predictions for pixels corresponding to the nearby ground plane are generally robust. This enables getting coarse estimate of the ground plane using RANSAC and detecting the number of pixels whose predicted depth projects them significantly below the ground. If that number is above a threshold, then the corresponding image subsequently ignored (we found a conservative threshold of 10 to work well in all our experiments, filtering out roughly 5% of the KITTI training dataset). During the second stage, we retrain the network on the subsampled dataset (from scratch to avoid the previous local optimum). As this subsampled dataset is de-biased, the network learns better depth estimates on dynamic objects. This process can be repeated, but we find that two stages enough to remove any traces of infinite depth in experiments, shown in Figure 3.
EXPERIMENTAL RESULTS
5.1 DATASETS
We use the standard KITTI benchmark (Geiger et al., 2013) for self-supervised training and evalu- ation. More specifically, we adopt the training, validation and test splits used in Eigen et al. (2014) with the pre-processing from Zhou al. (2017) to remove static frames, which is more suitable for
5

---

Published as a conference paper at ICLR 2020
Lower is Better
Higher is Better
Method
Superv.
Abs Rel
Sq Rel
RMSE RMSE
< 1.25
8 1.25* 8 1.25*
Garg et (2016)
M
0.152
1.226
5.849
0.246
0.784
0.921
0.967
Zou et al. (2018)
M
0.150
1.124
5,507
0.223
0.806
0.933
0.973
Godard et al. (2017)
M
0.141
1.186
5.677
0.238
0.809
0.928
0.969
Zhan et al. (2018)
M
0.135
1.132
5.585
0.229
0,820
0.933
0.971
Godard et al. (2018) (R18)
M
0.115
0.903
4,863
0.193
0,877
0.959
0.981
Godard ct al. (2018) (R50)
M
0.112
0.851
4.754
0.190
0.881
0,960
0,981
Guizilini
(2019) (MR)
M
0.108
0.727
4.426
0.184
0.885
0.963
0.983
Guizilini et al. (2019) (HR)
M
0.104
0.758
4.386
0.182
0.895
0,964
0.982
Casser et al. (2019)
S+Inst
0.141
1.025
5,290
0.215
0.816
0.945
0.979
Chen et al. (2019)
S+Sem
0.118
0.905
5.096
0.211
0.839
0,945
0.977
Ochs et al. (2019)
D+Sem
0.116
0.945
4.916
0.208
0,861
0,952
0.968
Ours (MR)
M+Sem
0.102
0.698
4.381
0.178
0.896
0.964
0.984
Ours (HR)
M+Sem
0.100
0.761
4.270
0.175
0.902
0.965
0.982
Table I: Quantitative performance comparison of our proposed architecture on KITTI for depths up to 80m. M refers to methods that train using monocular images, refers to methods that train using stereo pairs, D refers to methods that ground-truth depth supervision, Sem refers to methods that include semantic information, and Inst refers to methods that include semantic and instance information. MR indicates 640 192 input images, and HR indicates 1280 384 input images. Our proposed architecture able to further improve the current state of the art in self-supervised monocular depth estimation, and outperforms other methods that exploit semantic information (including ground truth labels) by substantial margin.
monocular supervuud learning. This results in 39810 images for training, 4424 for validation, and 697 for evaluation. Following common practice, we pretrain our depth and pose networks on the CityScapes dataset (Cordts et al., 2016), consisting of 88250 unlabeled images. Unless noted otherwise, input images are downsampled to 640 x 192 resolution and output inverse depth maps are upsampled full resolution using bilinear interpolation. Our fixed semantic segmentation network is pretrained on Cityscapes, achieving a mloU of 75% on the validation set.
5.2 IMPLEMENTATION DETAILS
We implement our models with PyTorrc (Paszke al., 2017) and follow the same training protocols of Guizilini et al. (2019) when optimizing our depth and pose networks. The initial training stage conducted the CityScapes dataset for 50 epochs, with a batch size of per GPU and initial depth and pose learning rates of 2 10-4 and 5 10-4 respectively, that are halved every 20 epochs. Afterwards, the depth and pose networks are fine-tuned on KITTI for 30 epochs, with the same parameters and halving the learning rates after every 12 epochs. This fine-tuning stage includes the proposed architecture, where information from the fixed semantic network, pretrained separately, is used to directly guide the generation of depth features. There is no direct supervision at any stage during depth training, all semantic information derived from the fixed secondary network.
When pretraining the semantic segmentation network, we use a ResNet-50 backbone with Imagenet (Deng et al., 2009) pretrained weights and optimize the network for 48k iterations on the Scayes dataset with learning rate of 0.01, momentum of 0.9, weight decay of 10-4, and batch size of 1 per GPU. Random scaling between (0.7, 1.3), random horizontal flipping, and a crop size of 1000 x 2000 are used for data augmentation. We decay the learning rate by factor of 10 at iterations 36k and 44k. Once training is complete, the semantic segmentation network is fixed and becomes the only source of semantic information when fine-tuning the depth and pose networks KITTI.
5.3 DEPTH ESTIMATION PERFORMANCE
Our depth estimation results are summarized Table I, where we compare our proposed architec- ture with other published works. From these results we can see that the introduction of semantically- guided geometric representation learning further improves upon the current state of the art self- supervised monocular depth estimation from Guizilini et al. (2019), which served our baseline. Our approach also outperforms other methods that leverage semantic information by a substantial margin, even those using ground-truth KITTI semantic segmentation and depth labels during train-

---

Published as a conference paper at ICLR 2020
Lower is Better
Higher is Better
Class-
Network
SEM
7ST
Abs Rel
Sq Rel
RMSE
RMSE
8 < 1.25
< 1.25
< 1.25
Abs Rel
0.120
0.896
4.869
0.198
0.868
0.957
0.981
0.149
ResNet-18
v
0.117
0.854
4.714
0.191
0.873
0.963
0.981
0.139
0.117
0.900
4.826
0.196
0.873
0.967
0.980
0.144
ResNet-50
0.113
0.831
4.663
0.189
0.878
0.971
0.983
0.136
0.108
0.727
4.426
0,184
0.885
0.963
0.983
0.132
PackNet
0.103
0.710
4.301
0.179
0.895
0.964
0.984
0.121
0.102
0.698
4.381
0.178
0.896
0.963
0.984
0.117
Table Ablative analysis our semantic guidance (SEM) and two-stage-training (TST) contri- butions. The last column indicates class-average Abs. Rel. obtained by averaging all class-specific depth errors in Figure 4, while other columns indicate pixel-average metrics.
ing (Ochs al., 2019). Furthermore, in Figure 5 also present qualitative results showing the improvements in depth estimation generated by our proposed framework, compared to our base- line. Note how semantically-guide architecture produces sharper boundaries and better object delineation, especially in structures further away or not clearly distinguishable in the input image.
5.4
ABLATIVE ANALYSIS
5.4.1 DIFFERENT DEPTH NETWORKS
To better evaluate main contribution, we provide ablative analysis showing how it generalizes to different depth networks. To this end, we consider two variations of the widely used ResNer architecture as the encoder for our depth network: ResNer-18 and ResNer-50 (the same pretrained semantic network was used in all experiments). Depth estimation results considering these variations are shown in Table 2, where we can see that our proposed semantically -guided architecture is able to consistently improve the performance of different depth networks, for all considered metrics.
5.4.2
DEPTH PERFORMANCE
To further showcase the benefits of semantically-guit guidded architecture, also provide class- specific evaluation metrics, as shown Figure 4. As we do not have ground-truth semantic segmen- tation for these images, we use the prediction of the semantic network bin pixels per predicted category, and evaluate only on those pixels. From these results we can see that our proposed archi- tecture consistently improves depth performance for pixels across all predicted classes, especially those containing fine-grained structures and sharp boundaries, e.g. poles and traffic signs.
0.25
Guizilin et al (2019) (MR)
0.2
Abs. Rel.
0.15
0.1
0.05
Road Sidewalk Building
Wall
Fence
Pole
T. Light
T. Sign
Vegetation
Terrain
Person
Rider
Car
Truck
Bus
Train
Motorcycle
Bicycle
Average
Figure 4: Class-specific depth evaluation for our proposed architecture (blue), relative to our base- line (red). The rightmost column indicates class-average depth metrics, obtained by averaging all individual classes. The introduction of semantically -uiidd features, in conjunction with our pro- posed two-stage training methodology to address the infinite depth problem, consistently improved depth results for all considered classes (lower is better).

---

Published as a conference paper at ICLR 2020
Figure 5: Qualitative results of our proposed architecture. The left, middle, and right columns show respectively input images, baseline predicted depth maps (Guizilini et al., 2019), and the depths maps obtained using our proposed architecture. Our semantic-aware depth network predicts sharper boundaries and fine- tne-grained details distant objects. The dotted lines indicate class- dass-av errors, obtained by averaging all the class-specffie depth errors.
We also measure the impact of our two-stage training process, which is expected to address the infinite depth problem in dynamic objects. Although we find the pixel-average difference perfor- mance to not be significant (see Table 2), there is significant improvement in class-average depth estimation, from 0.121 to 0.117 Abs-Rel. This is because the number of pixels affected by the in- finite depth problem is vastly smaller than the total number of pixels. However, when considering class-average depth evaluation, the improvement over classes such as cars (0.200 to 0.177 Abs-Rel) and motorcycles (0.091 to 0.069) becomes statistically significant. This further exemplifies the im- portance of fine- fine-graine grained metrics in depth evaluation, so these underlying behaviors can be properly observed and accounted for in the development new techniques.
CONCLUSION
This paper introduces novel architecture for self-supervised monocular depth estimation that lever- ages semantic information from fixed pretrained network to guide the generation of multi-level depth features via pixel-adaptive convolutions. Our monodepth network learns semantic-aware ge- ometric representations that can disambiguate photometric ambiguities in self-supervised learning structure-fron fooe-ffroomoooon context. Furthermore, we introduce a two-stage training process that resam- ples training data to overcome common bias dynamic objects resulting in predicting them at infinite depths. Our experiments on challenging real-world data shows that our proposed architec- ture consistently improves the performance of different monodepth architectures, thus establishing a new state of the art in self-supervised monocular depth estimation. Future directions of research include leveraging other sources of guidance (i.e. instance masks, optical flow, surface normals), as well as avenues for self-supervised fine-tuning of the semantic network.
8

---

Published as a conference paper at ICLR 2020
REFERENCES
Min Bai, Wenjie Luo, Kaustav Kundu, and Raquel Urtasun. Exploiting semantic information and deep matching for optical flow. In ECCV, 2016.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. CoRR, 2019. 13, 14
Vincent Casser, Soeren Pirk, Reza Mahjourian, and Anelia Angelova. Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos. AAAI, 2019. 2, 3, 5, 6
Po-Yi Chen, Alexander H. Liu, Yen-Cheng Liu, and Yu-Chiang Frank Wang. Towards scene under-
standing: Unsupervised monocular depth estimation with semantic-aware representation. In The
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 1, 6
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network
learning by exponential linear units (elus). ICI.R, 2016.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016. 12
Jia Deng, Wei Dong, Richard Socher, Li jia Li, Kai Li, and Li Fei-fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 2009, 6, 12
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using a multi-scale deep network. In Advances in neural information processing systems, 2014. 1, 2,
Huan Fu, Mingming Gong. Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2002-2011, 2018. 2
Ravi Garg, Vijay Kumar BG, Gustavo Carneiro, and Ian Reid. Unsupervised enn for single view depth estimation: Geometry to the rescue. In European Conference Computer Vision, pp- 740-756. Springer, 2016. 2,
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11))231-12337 2013. 1,5
Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estima- tion with left-right consistency. In CVPR, volume 2, pp. 2017. 3,
Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self- supervised monocular depth prediction. arXiv:1806.01200 2018. 5, 14
Vitor Guizilini, Sudeep Pillai, Rares Ambrus, and Adrien Gaidon. Packnet-sfm: packing for self-supervised monocular depth estimation. arXiv preprint arXiv:1905.02693, 2019. 1, 2, 3, 6, 8, 14
Fatma Güney and Andreas Geiger. Displets: Resolving stereo ambiguities using object knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR). 2015.
Xiaoyang Guo, Hongsheng Li, Shuai Yi, Jimmy Ren, and Xiaogang Wang. Learning monocular depth by distilling cross-domain stereo networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 484-500, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.7

---

Published as a conference paper at ICLR 2020
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Ad- vances in neural information processing systems, pp. 2017-2025, 2015. 2
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 7482-7491, 2018.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 6399-6408, 2019. 5
Maria Klodt and Andrea Vedaldi. Supervising the new with the old: Learning sfmm from sfm. In European Conference on Computer Vision, pp. 713-728. Springer, 2018. 2
Jae-Han Lee, Minhyeok Heo, Kyung-Rae Kim, and Chang-Su Kim. Single-image depth estimation based on fourier domain analysis. International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 330-339, 2018.
Kuan-Hui Lee, German Ros, Jie Li, and Adrien Gaidon. Spigan: Privileged adversarial learning from simulation. In ICLR, 2019.
Bo Li, Chunhua Shen, Yuchao Dai, Anton Van, and Mingyi He. Depth and surface normal estimation from monocular images using regression on deep features and hierarchical crfs. In International
Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1119-1127. 2015. 2
Jie Li, Allan Raventos, Arjun Bhargava, Takaaki Tagawa, and Adrien Gaidon. Learning to fuse things and stuff. arXiv preprint arXiv: 2018. 5
Zhizhong Li and Derek Hoiem. 2017. 3
Learning without forgetting. IEEE TPAMI, 40(12):2935-2947,
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017. 5
Reza Mahjourian, Martin Wicke, and Anelia Angelova. Unsupervised learning of depth and ego- motion from monocular video using 3d geometric constraints. In Proceedings of the IEEE Con- ference Computer Vision and Patlern Recognition, pp. 5667-5675, 2018. 2
Fabian Manhardt, Wadim Kehl, and Adrien Gaidon. Roi-10d: Monocular lifting of 2d detection to 6d pose and metric shape. International Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Jeff Michels, Ashutosh Saxena, and Andrew Y Ng. High speed obstacle avoidance using monoc- ular vision and reinforcement learning. Proceedings of the 22nd international conference on Machine learning, pp. 5-3-55330 600. ACM, 2005. 1
Matthias Ochs, Adrian Kretz, and Rudolf Mester. Sdnet: Semantically guided depth estimation
network. In 19Xiv:I907.1004 10659, 2019. 1. 6.7
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPN-WWWW W. 2017.
Sudeep Pillai, Rares Ambrus, and Adrien Gaidon. Superdepth: Self-supervised, super-resolved monocular depth estimation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2019.
Lorenzo Porzi, Samuel Rota Bulo, Aleksander Colovic, and Peter Kontschieder. Seamless scene segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- fion, pp. 8277-8286, 2019.
Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia. Geonet: Geometric neural network for joint depth and surface normal estimation. In International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 283-291, 2018. 2
10

---

Published as a conference paper at ICLR 2020
Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik Learned-Milles and Jan Kautz. Pixel- adaptive convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 1,
Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovit- skiy, and Thomas Brox. Demon: Depth and motion network for learning monocular stereo. In IEEE Conference on computer vision and pattern recognition (CVPR), volume 5. pp. 6, 2017. 2
Chaoyang Wang, José Miguel Buenaposada, Rui Zhu, and Simon Lucey. Learning depth from monocular videos using direct methods. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 2022-2030, 2018.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 2004. 2, 3
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the 15th European Conference on Computer Vision (ECCV), pp. 3-19, 2018. 4
Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: unified panoptic segmentation network. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp.
2019. 5
Nan Yang, Rui Wang. Jörg Stückler, and Daniel Cremers. Deep virtual stereo odometry: Leveraging deep depth prediction for monocular direct sparse odometry. arXiv: 1807.0250, 2018. 2
Zhenheng Yang, Peng Wang, Wei Xu, Liang Zhao, and Ramakant Nevatia. Unsupervised learning of geometry with edge-aware depth-norma consistency. arXiv preprint arXiv:1711.03665, 2017.
Zhichao Yin and Jianping Shi. Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, 2018. 2
Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian Reid. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 340-349, 2018. 6
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, volume 2, pp. 7, 2017. 2, 3,5
Yuliang Zou, Zelun Luo, and Jia-Bin Huang. Df-net: Unsupervised joint learning of depth and flow using cross cross-task consistency. In European Conference on Computer Vision, 2018. 1, 2, 6

---

Published as a conference paper at ICLR 2020
A PRE-TRAINING THE SEMANTIC SEGMENTATION NETWORK
The introduction of a semantic segmentation network to the depth estimation task increases the depth estimation performance, however it also increases model complexity (e.g. number of trainable parameters). To investigate that the increased performance for the depth estimation task is indeed due the semantic features encoded in the secondary network, we perform in-depth analysis (summarized in Table 3) where we explore the impact of pre-training the semantic segmentation network before it is used to guide the generation of depth features. From these results we can see that the presence of semantic information encoded in the secondary network indeed leads to an increase in performance, and that fine-tuning this secondary network for the speficic task of depth estimation actually decreases performance.
In the first two rows an untrained semantic network is utilized, with only its encoder initialized from ImageNet (Deng et al., 2009) weights. Two different scenarios are explored: in the first one (D) only the depth network is fine-tuned a self-supervised fashion, while in D+S both networks fine-tuned together in the same way. As expected, using untrained features guidance leads to significantly worse results, since there is no structure encoded in the secondary network and the primary network needs learn filter out all this spurious information. When both networks fine-tuned simultaneously. results improve because now the added complexity from the secondary network can be leveraged for the task of depth estimation, however there is still no improvement over the baseline.
Next, the semantic network was pre-trained on only half of the CityScapes (Cordts al., 2016) dataset (samples chosen randomly), leading to a worse semantic segmentation performance (vali- dation mloU of around 70% vs. 75% for the fully trained one). This partial pre-training stage was enough to enable the transfer of useful information between networks, leading to improvements over the baseline. Interestingly, fine-tuning both networks for the task of depth estimation actually hurt performance this time, which we attribute to forgetting the information contained in the sec- ondary network, as both networks are optimized for the depth task. When the semantic network is pretrained with all of CityScapes (last two rows), these effects are magnified, with fine-tuning only the depth network leading to our best reported performance (Table 1) and fine-tuning both networks again leading to results similar to the baseline.
Lower is Better Sq Rel
Higher is Better
Method
Pre-Train
Fine-Tume
Abs Rel
RMSE RMSE,
6 < 1.25
< 1.25? ó < 1.25
Baseline
D
0.108
0.727
4.426
0.184
0.885
0.963
0.983
I
D+S
0.116
0.847
4.751
0.192
0.879
0.960
0.981
I
D
0.197
1.323
6.114
0.265
0.776
0,918
0.966
Proposed
CS (1/2)
D+S
0.109
0.737
4.389
0.185
0.884
0.962
0,982
CS (1/2)
D
0.104
0.716
4.322
0.180
0.893
0.964
0.984
CS
D+S
0.107
0.741
4.407
0.183
0.883
0.963
0.983
CS
D
0.102
0.698
4.381
0.178
0.896
0.964
0.984
Table 3: Analysis of the impact of pre-training the semantic segmentation network. On the Pre- Train column, I indicates ImageNet (Deng et al., 2009) pretraining and CS indicates CityScapes (Cordts et al., 2016) pretraining, with 1/2 indicating the use of only half the dataset (samples chosen randomly). In the Fine-Ture column, D indicates tine-tuning the depth network and S indicates fine- tuning the semantic network (note that this is self-supervised fine-tuning for the depth task, using the objective described in Section 3.1).
B UNCERTAINTY AND GENERALIZATION TO DIFFERENT OBJECTS
In a self-supervised setting, increasing the number of unlabeled videos used for depth training is expected to lead to an increasing specialization away from the domain in which the semantic net- work was pre-trained. This might result in harmful guidance if our method not robust to this gap. However, our approach does not use semantic predictions directly, but rather the decoded features of the semantic network themselves, which represent general appearance information that should be more robust to this domain gap. To validate our hypothesis, further explore the impact
12

---

Published as a conference paper at ICLR 2020
of erroneous semantic information in the performance of our proposed semantically-guiie depth framework. Figure we present qualitative results highlighting situations in which our pretrained semantic network failed to generate correct semantic predictions for certain objects in the scene, and yet our proposed framework was still able to properly recover depth values for that portion of the environment. These exemplify possible scenarios for erroneous semantic prediction.
Imprecise boundaries: in the first row, we can see that the semantic segmentation network does not correctly detect the traffic sign, yet the semantically-guice depth network predicts its shape and depth accurately.
Wrong classification: in the second row, the truck was mistakenly classified as partially "road" and "'bildding", however our semantically-gui guliyyggggdd depth network was still able to properly recover its overall shape with sharp delineation that was not available from its se- mantic contour. A similar scenario happens in the same image, with "fence" being partially labeled as "bicycle".
Missing ontology: there is "trash can" class the CityScapes ontology, however in the third row our semantically-guided depth network was able to correctly reconstruct such object even though it was classified "fence", similarly its surroundings.
Object Hallucination: in the fourth row, the contour of "person" was erroneously intro- duced in the image and correctly removed by our semantically-guide framework.
These examples are evidence that our proposed framework is able to reason over the uncertainty inherent to semantic classification, leveraging this information when accurate achieve the results reported in this paper, but also discarding if necessary to generate better reconstruction according to the self-supervised photometric loss.
DHL
Figure 6: Examples of erroneous semantic predictions that still led to accurate depth predictions using our proposed semantically -auiidd depth framework.
C GENERALIZATION TO DIFFERENT DATASETS
In the previous sections, we show that our proposed framework is robust degraded semantic network, both by pretraining the semantic network with fewer annotated labels (Appendix A) and also by providing evidence that the depth network able reason over erroneous predictions to still generate accurate reconstructions (Appendix B). We now go one step further and analyze how our proposed semantically-guii guiided framework generalizes to a dataset that was used neither during pre-training nor for fine-tuning, To this end, we evaluate our KITTI depth model the recently released NuScenes dataset (Caesar et al., 2019). The official NuScenes validation split is used, containing 6019 images from the front camera with ground-trute -tuth depth maps generated by LiDAR reprojection. Results presented in Table provide additional evidence that our method indeed re- sults in generalization improvements, even significantly different data from different platforms
13

---

Published as a conference paper at ICLR 2020
and environments (Karlsruhe, Germany for KITTI Boston, USA and Singapore for NuScenes), outperforning the state of the art methods and our baseline (Guizilini et al., 2019).
Method
Abs Rel
Sq Rel
RMSE
RMSBleg
< 1.25
6 < 1.25
< 1.25
Godard et al. (2018) (R18)
0.212
1.918
7.958
0.323
0.674
0.898
0,954
Godard et al. (2018) (R50)
0.210
2.017
8.111
0.328
0.697
0,903
0,960
Guizilini et al. (2019) (MR)
0.187
1.852
7.636
0.289
0.742
0.917
0.961
Ours (MR)
0.181
1.505
7.237
0.271
0.765
0.931
0.969
Table 4: Generalization capability of different networks, trained on both KITTI and CityScapes datasets and evaluated on the NuScenes (Caesar al., 2019) dataset. Our proposed semantically- guided architecture is able to further improve upon the baseline from Guizilini et al. (2019), which only used unlabeled image sequences for self-supervised depth training.
14


Reviews:
[{"review_id": "rkxuEDryqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The underlying problem considered in this manuscript is inferring depth from geometry of two dimensional images. The novelty here is to integrate a semantic classification model along with depth inference. It is a challenging problem and a neat idea to pursue. The paper is well-written and easy to follow. \nHowever, the empirical work in the paper is not persuasive. Table 1 contains results whose significance is hard to judge. What does a RMSE difference of 2.3 mean in the context of depth estimation? Table 1 carries no uncertainty in results which is just not acceptable in a setting that has several sources of uncertainty. Similarly, Fig 4 shows there is advantage in the use of the pre-trained semantic network, it is not clear if this difference is significant. And I also think consideration should be given to the fact that in a deployment setting, new objects not previously seen in the semantic categories (UFO) may appear and one ought to understand if the semantic network might decrease performance (because of the unseen class). Hence I think which the idea advanced in the paper has merits, the manuscript is not really ready for publication.  ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkgdigaTFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a using pixel-adaptive convolutions to leverage semantic labels in self-supervised monocular depth estimation. The semantic features are predicted by a pretrained network rather than relying on a ground truth. Moreover, a two-stage training process in proposed in order to filter out images leading to erroneous SfM predictions. The method is evaluated with different networks on the KITTY dataset.\n\nThe paper is very well written and clear. The applications of per-pixel convolutions to this problem seems sound and the experimental validation seems satisfactory. I have however one main concern (1) and a few additional questions below:\n\n1) While (Guizilini 2019) shows that using a larger set of unannotated videos and allows the self-supervised method to eventually outperform supervised methods, this study is not done here. This makes me question the applicability of the approach, as using large unlabelled videos would probably lead to noisy segmentations that could be unhelpful to the depth estimation. Showing an improvement over the supervised baseline would be a much stronger experimental validation, as for now it is difficult to know exactly why in which scenario this method should be used, rather than a supervised network or vanilla packnet.\n\n2) I see that you obtain the same numbers in Table 2 / PackNet / row 1 as in (Guizilini 2019); I would like to confirm that you used exactly their self-objective loss, in all your experiments? I would suggest adding to section 3.1. the fact the fact that the loss is the same is in (Guizlini 2019), as a reader could assume that there is novelty in the loss formulation.\n\n3) Have you tried fine-tuning the whole architecture including the semantic network end-to-end?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HyeZcWk9Kr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes to leverage a pre-trained semantic segmentation network to learn semantically adaptive filters for self-supervised monocular depth estimation. Additionally, a simple two-stage training heuristic is proposed to improve depth estimation performance for dynamic objects that move in a way that induces small apparent motion and thus are projected to infinite depth values when used in an SfM-based supervision framework. Experimental results are shown on the KITTI benchmark, where the approach improves upon the state-of-the-art.\n\nOverview:\n\n+ Good results\n+ Doesn't require semantic segmentation ground truth in the monodepth training set\n\n- Not clear if semantic segmentation is needed\n- Specific to street scenes\n- Experiments only on KITTI\n\nThe qualitative results look great and the experiments show that semantic guidance improves quantitative performance by a non-trivial factor. The qualitative results suggest that the results produced with semantic guidance are sharper and more detailed. However, it is not clear that using features from a pre-trained semantic segmentation network is necessary. The proposed technical approach is to use the pixel-adaptive convolutions by Su et. al. to learn content-adaptive filters that are conditioned on the features of the pre-trained semantic segmentation network. These filters could in principle be directly learned from the input images, without needing to first train a semantic segmentation network. The original work by Su et. al. achieved higher detail compared to their baseline by just training the guidance network jointly.  Alternatively, the guidance network could in principle be pre-trained for any other task. The main advantage of the proposed scheme is that the guidance path doesn't need to be trained together with the depth network. On the other hand, unless shown otherwise, we have to assume that the network needs to be pre-trained on some data that is sufficiently close to the indented application domain. This would limit the approach to situations where a reasonable pre-trained semantic segmentation network is available.\n\nThe proposed heuristic to filter some dynamic objects is very specific to street scenes and to some degree even to the KITTI dataset. It requires a dominant ground plane and is only able to detect a small subset of dynamic motion (e.g. apparent motion close to zero and object below the horizon). It is also not clear what the actual impact of this procedure is. Section 5.4.2 mentions that Abs. Rel decreases from 0.121 to 0.119, but it is not clear to what this needs to be compared to as there is no baseline in any of the other tables with an Abs. Rel of 0.121. Additionally, while the authors call this a minor decrease, the order of magnitude is comparable to the decrease in error that this method shows over the state-of-the-art (which the authors call statistically significant) and also over the baselines (c.f. Table 2). Can the authors clarify this?\n\nRelated to being specific to street scenes: The paper shows experiments only on the KITTI dataset. The apparent requirement to have a reasonable semantic segmentation model available, make it important to evaluate also in other settings (for example on an indoor dataset like NYU) to show that the approach works beyond street scenes (which is one of the in practice not so interesting settings for monocular depth estimation since it is rather easy to just equip cars with additional cameras to solve the depth estimation problem).\n\nNeed for a reasonable segmentation model: It is not clear in how far the quality of the segmentation network impacts the quality for the depth estimation task. What about the domain shift where the segmentation model doesn't do so well? Even if the segmentation result is not used directly, the features will still shift. How much would depth performance suffer?\n\nSummary:\nWhile the results look good on a single dataset, I have doubts both about the generality of the proposed approach as well as the need for the specific technical contribution.\n\n=== Post rebuttal update ===\nThe authors have addressed many of my initial concerns and provided valuable additional experimental evaluations. While I'd like to upgrade my recommendation to weak accept, I strongly encourage the authors to provide additional experiments on different datasets (at least NYU). ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a using pixel-adaptive convolutions to leverage semantic labels in self-supervised monocular depth estimation. Although there were initial concerns of the reviewers regarding the technical details and limited experiments, the authors responded reasonably to the issues raised by the reviewers. Reviewer2, who gave a weak reject rating, did not provide any answer to the authors comments. We do not see any major flaws to reject this paper.
Decision: Accept (Poster)