Pseudo Label-Guided Multi Task Learning for Scene Understanding
Authors: Sunkyung Kim
Hyesong Choi
Dongbo Min
Venue:  (2021)
Keywords: Multi-task learning
monocular depth estimation
semantic segmentation
pseudo label
cross-view consistency

Abstract: Multi-task learning (MTL) for scene understanding has been actively studied by exploiting correlation of multiple tasks. This work focuses on improving the performance of the MTL network that infers depth and semantic segmentation maps from a single image. Specifically, we propose a novel MTL architecture, called Pseudo-MTL, that introduces pseudo labels for joint learning of monocular depth estimation and semantic segmentation tasks. The pseudo ground truth depth maps, generated from pretrained stereo matching methods, are leveraged to supervise the monocular depth estimation. More importantly, the pseudo depth labels serve to impose a cross-view consistency on the estimated monocular depth and segmentation maps of two views. This enables for mitigating the mismatch problem incurred by inconsistent prediction results across two views. A thorough ablation study validates that the cross-view consistency leads to a substantial performance gain by ensuring inference-view invariance for the two tasks.

Under review as a conference paper at ICLR 2021
PSEUDO LABEL-GUIDED MULTI TASK LEARNING FOR SCENE UNDERSTANDING
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-task learning (MTL) for scene understanding has been actively studied by exploiting correlation of multiple tasks. This work focuses on improving the per- formance of the MTL network that infers depth and semantic segmentation maps from a single image. Specifically, we propose novel MTI. architecture, called Pscudo-MTL, that introduces pseudo labels for joint learning of monocular depth estimation and semantic segmentation tasks. The pseudo ground truth depth maps, generated from pretrained stereo matching methods, are leveraged to supervise the monocular depth estimation. More importantly, the pseudo depth labels serve impose cross-view consistency mentation maps of two views. This enables for mitigating the mismatch problem incurred by inconsistent prediction results across two views. A thorough ablation study validates that the cross-view consistency leads to substantial performance gain by ensuring inference-view invariance for the two tasks.
the estimated monocular depth and seg-
INTRODUCTION
Scene understanding has become increasingly popular in both academia and industry essen- tial technology for realizing a variety of vision-based applications such as robotics and autonomous driving. 3D geometric and semantic information of scene often serve as a basic building block for high-level scene understanding tasks. Numerous approaches have been proposed for inferring depth map (Garg et al. 2016; Godard al.] 2019) or grouping semantically similar parts (Chen iet al., 2017 Yuan et al. 2019 from a single image. In parallel with such rapid evolution for individual tasks, several approaches (Chen et al. 2019 Zhang al., 2018 Guizilini al. 2020b Liu et al.) 2019) have focused on boosting the performance through joint learning of the semantic segmentation and monocular depth estimation tasks by considering that the two tasks are highly cor- related. For instance, pixels with the same semantic segmentation labels within an object are likely to have similar (or smoothly-varyinn the boundary of two objects containing different semantic segmentation labels. These properties have been applied to deep networks to enhance the semantic segmentation and monocular depth estimation tasks in a synergetic manner.
depth values. An abrupt change depth values often implies
In (Chen et al. 2019), they proposed a joint learning model that learns semantic-aware represen- tation to advance the monocular depth estimation with the aid of semantic segmentation. A depth map is advanced by making use of loss functions designed for the purpose of bonding geometric and semantic understanding. The method in GGuizilini et al. 20206) proposed a new architecture that improves the accuracy of monocular depth estimation through the pixel-adaptive convolution (Su al. 2019) using semantic feature maps computed from pre-trained semantic segmentation networks. Despite the improved monocular depth accuracy over single monocular depth network, the performance improvement the semantic segmentation task by the aid of geometrical repre- sentation has not been verified (Chen al. 2019), or even the semantic segmentation network was fixed with pretrained parameters (Guizilini et al. 2020b).
A generic computational approach for multi-task learning (MTL) was proposed in (Zamir et al. 2018). which models the structure across twenty six tasks, including 2D, 2.5D, 3D, and semantic tasks, by finding first and higher-order transfer learning dependencies across them in latent space to seamlessly reuse supervision among related tasks and/or solve them in a single network without increasing the complexity significantly. This was further extended by imposing a cross-task consis-

---

Under review as a conference paper at ICLR 2021
tency based on inference-per -ath invariance graph of multiple tasks (Zaiir et al. 2020). Though these approaches provide a generie and principled way for leveraging redundancies across multiple tasks, there may be limitations to improving the performance of individual tasks in that it is difficult to consider task-specific architectures and loss functions in such unified frameworks. With the same objective yet with different methodology, the method in (Liu et al., 2019) proposes a novel MTT. architecture consisting of task-shared and task-specific networks based on task-attention modules, aiming learn both generalizable features for multiple tasks and features tailored each task. They validated the performance in the joint learning of monocular depth and semantic segmentation.
In this paper, we propose novel MTL architecture for monocular depth estimation and semantic segmentation tasks, called pseudo label-guided multi-task learning (Pseudo-MTL). The proposed ar- chitecture leverages geometrically- and semantically-guidee representations by introducing pseudo ground truth labels. When a pair of stereo images is given as inputs, our method first generates pseudo ground truth left and right depth maps by using existing pre-trained stereo matching net- works (Pang et al., 2017 Chang & Chen 2018). To prevent inaccurate depth values from being used, a stereo confidence map (Poggi & Mattoccial 2016) is used together as auxiliary data that measures the reliability of the pseudo depth labels. These leveraged for supervising the monoc- ular depth network, obtaining substantial performance gain over recent self-supervised monocular depth estimation approaches (Godard al., 2017 2019). More importantly, the pseudo depth la- bels are particularly useful when imposing a cross-view consistency across left and right images. The estimated monocular depth and segmentation maps of two views are tied from a geometrie per- spective by minimizing the cross-view view consistency loss, alleviating the mismatch problem incurred by inconsistent prediction across two views significantly. We will verify through an intensive ab- lation study that the proposed cross-consistency loss leads to substantial improvement on both tasks. Experimental results also show that our approach achieves an outstanding performance over state-of-the-arts. In short, our novel contributions can be summarized as follows.
We propose a novel MTL approach that jointly performs monocular depth estimation and semantic segmentation through pseudo depth labels.
The cross-view consistency loss based on the pseudo depth labels and associated confidence maps is proposed to enable consistent predictions across two views.
An intensive ablation study is provided to quantify the contribution of the proposed items to performance improvement.
RELATED WORK
Monocular Depth Estimation While early works for monocular depth estimation are based on supervised learning, self-supervised learning has attracted increasing interest in recent approaches (Godard et al. 2017] 2019, Watson et al., 2019) to overcome the lack of ground truth depth labels. Here, we review works mostly relevant to our method. Godard et al. (Godard et al.) [017]] 2019) pro- posed the deep network that infers a disparity map using the image reconstruction loss and left-right consistency loss from pair of stereo images monocular videos. Chen al. (Chen et al.) 2019) in- fers both disparity and semantic segmentation maps by enforcing the cross consistency across stereo images address the mismatch problem of (Godard et al.) 2017). Several approaches have focused on improving the monocular depth estimation through the aid of segmentation networks, e.g., by stitching local depth segments from instance segmentation with respect to scale and shift (Wang et al.) 2020) or leveraging pretrained semantic segmentation networks to guide the monocular depth estimation (Guizilini et al. 2020b).
Semantic Segmentation A deep convolutional encoder-decoder architecture for semantic segmen- tation proposed in (Badrinarayanan et al.. 2017) has been widely used as backbone. The pyramid pooling module was proposed for leveraging global context through aggregation of different region- based contexts (Zhao et al., 2017). Some segmentation works attempted to combine different tasks to improve segmentation performance. Gated- (Takikawa et al. 2019) refines segmentation features and boundary features. FuseNet (Hazirbas et al., 2016)
results by fusing semantic-regee proposed to fuse features from color and depth images for improving the segmentation performance.
Multi-task learning In (Chen et al. 2019 Takikawa et al. 2019 Zhang et al. 2018), they proposed to leverage task-specific loss functions to tie up two (or more) tasks within the MTL. architecture. For
2

---

Under review as a conference paper at ICLR 2021
Dy
d
E
D,
D2
E
(b)
Figure 1: Network architecture: (a) Pseudo MTL based on the prroder-decode,, (b) To impose the cross-view consistency, is applied for left and right images, respectively.
instance, Chen et al. (Chen et al. 2019) attempted to improve monocular depth accuracy by using the loss functions that measure the consistency between geometric and semantic predictions. The generic computational approach for MTL was proposed by leveraging redundancies across multiple tasks in a latent space in (Zamir et al. 2018 2020). The task-attention modules were introduced to extract features for individual tasks in (Misra et al. 2016 Liu al., 2019 Tha et al. (020). In this work, we focus on improving the performance of the MT.. architecture for monocular depth esti- mation and semantic segmentation tasks by using the cross-view consistency loss based on pseudo labels.
3 PROPOSED METHOD
3.1 OVERVIEW AND ARCHITECTURE DESIGN
Our Pseudo-MTL approach focuses on improving the performance of the monocular depth estima- tion and semantic segmentation tasks through task-specific losses defined based the pseudo depth labels generated by using pre-trained stereo matching networks (Pang et al., 2017). The stereo con- fidence maps used together auxiliary data to compensate for estimation errors in the pseudo depth labels. These are effective in mitigating undesired artifacts of errors that may exist in the pseudo depth labels. In work, chose the CCNN (Poggi & Mattoccia 2016) for calculating the confidence map. but more advanced confidence estimation approaches can also be used.
As shown Figure [ the proposed Pscudo-MTL network is based on the encoder-decoder archi- tecture, in which single encoder takes an image and two decoders predict the monocular depth and semantic segmentation maps. The encoder network consists of the convolutional layers of the VGG network (Simonyan & Zisserman 2013). Two decoders, Da for monocular depth estimation and D, for monocular depth estimation, are designed symmetrically with the encoder. While two tasks share the encoder, the task-specific decoder branches are used for each task.
The pseudo depth labels and the segmentation label maps of stereo images are used for supervising the proposed architecture. The monocular depth and segmentation maps of left and right images are estimated by passing each image to the proposed architecture, as shown in Figure [ The cross-view consistency loss is then imposed on the prediction results of two views. To be specific, the estimated monocular depth maps of left and right images are warped and tested using the pseudo depth labels for ensuring inference-vis view invariance the monocular depth estimation, and similar procedure also applied to the semantic segmentation.
Using the pseudo depth labels for training the proposed model is advantageous various aspects. The pseudo depth labels stereo images, filtered out by its confidence map. provides a better super- vision (Choi al. 2020 than recent self-super monocular depth estimation approaches. More importantly, the cress-view consistency based on the pseudo depth labels mitigates the mismatch problem by inconsistent prediction results of views, leading substantial performance gain. Our method aims at advancing the two tasks via task-specific losses based on pseudo ground truth labels, and existing MTL architectures, c.g. based on task-specific attention modules and adaptive balancing (Liu et al. 2019 Jha et al. 2020), can be used complementarily with our loss functions.
3.2 LOSs FUNCTIONS
Loss functions are divided into two parts, 1) supervised loss for depth and segmentation networks and 2) pseudo depth-guided reconstruction loss for cross-view consistency. Note that the supervised
3

---

Under review as a conference paper at ICLR 2021
loss used for monocular depth estimation relies on the pseudo depth labels generated from pair of stereo images.
3.2.1 LOSS FOR MONOCULAR DEPTH AND SEMANTIC SEGMENTATION
Depth maps d, for {l, r} predicted by the decoder Da for monocular depth estimation, are used for measuring the depth regression loss La as follows:
La Σ im (1.r)
Leeg(ei, di, d where Leeg di dRtt)
1
Ee(())
cx()) (di(p) de' (p)
Z
PE#
(1)
where 6N and dea indicate the confidence map and pseudo ground truth depth map of left (i 1) or
right (i r) images, respectively. The loss normalized with Z; =Ep (p). . represents a set of
all pixels. The confidence map serves to exclude inaccurate depth values of dfst when calculating the depth regression loss T.g. This can be used in various ways, including the hard thresholding (Cho al. 2019 [Tonioni et al. (020)) and the soft thresholding (Choi et al.] 2020). Among them, the soft thresholded confidence map (Choi et al., 2020)) is shown to be effective in the monocular depth estimation. Our work chose to threshold the confidence map through the soft-thresholding of (Choi al. 2020). We found that the pretrained threshold network already provides satisfactory results, and thus it was fixed during our network training.
A supervised loss for semantic segmentation is defined with the standard cross-entropy H:
Lg = Σ 5()
(1.))
8 and se denote the segmentation map, predicted by the decoder D, for semantic segmentation, and ground truth segmentation map, respectively. The supervised loss for both tasks is defined as Ls LaLa + a,L, with loss weights ag and 0'g.
3.2.2 CROSS-VIEW CONSISTENCY LOSS
Minimizing the supervised loss Lg for an individual view may often lead to the mismatched problem in the predicted depth and segmentation maps due to the lack of consistency constraints across two views. We address this issue by imposing the cross-view consistency across left and right images with the pseudo depth labels. Figure[]shows the procedure of computing the cross-view consistency losses with pseudo depth labels. The cross-view consistency loss for the monocular depth estimation defined as follows:
Ld,e -.dtLLaii + aasLa, + acd La,r.
(3)
Lait Lvog (a, di, G(dy; d(dr:) + Treg (cr, G(d;; dest), dr),
(4)
La,1 = Lreg dest, G(d,; dfg1) + Lieg (q, di, Giffet : .dfst)) La,r Leeg (cr, G(d;; d dpst) + Lireg (cr. G(d dPst), drr)
(5)
(6)
where ad,tr, Od, and Odd, denote weights for each loss. G(a; b) indicates the result of warping with depth map into another view. For instance, G(dy; d returns the depth map warped onto the left image using dpst. Ta,ir measures the cross-view consistency between two predicted depth maps dj and d,. Note that the warping function G is applied to d, and d,, respectively. Similar to the depth regression loss La, the confidence map used together prevent inaccurate values in the pseudo depth labels from being used. La, denotes the cross-view consistency for (dpst d,) and (dl, dyst) using the left pseudo label dpst. This implies that when warping d, (or dest) into the left image, the warped result should be similar to dpst (or di). La,- is defined a similar manner.
4

---

Under review as a conference paper at ICLR 2021
The cross- view consistency can also be applied semantic segmentation as follows:
Tyg,c Os Lastr + as Tag + O.s
Ligrr = q: H (st G(s,: pott )} cr H (G(s); d
(8)
( )) + q H H(s, G(sf;; ;d()))) Ls,1 = H (5] G( of
(9)
Ls,r c (G(: dPtt, s + ((((s dest), ),)))
(10)
where '.' indicates an element-wise multiplication. The confidence maps e and G. are also used to compensate for errors the pseudo depth labels dfttt and dpse, Note that for some training datasets that provide no ground truth segmentation maps, we generate pseudo ground truth segmentation maps. More details are provided in Section 3.3.
Note that in (Chen et al. (2119) the consistency for left and right segmentation maps consid- ered e.g., by minimizing H (st, G(s,; d:)) Two segmentation maps 8/ and 8, are aligned with the estimated monocular depth map d, How- ever, di is continuously updated during work training, and thus this may result in inac- curate alignments at early stage, often leading to divergences of loss. For these reasons, mini- mizing the loss H with respect to both monocu- lar depth and segmentation maps often becomes very challenging, and the performance gain by the consistency loss is relatively marginal. Con- trarily, our approach is more effective in impos- ing the cross-view consistency that more accurate pseudo depth labels, obtained from stereo matching networks, are used, 2) the con- fidence map helps to filter out inaccurate depth values in the pscudo ground truth depth maps. Furthermore, we extend the cross-vicw view consis- tency to the monocular depth estimation, which is infeasible in the recent superviud monocular depth estimation approaches (Godard et al., 2017 2019 Watson et al. 2019) that rely on the reconstruction loss only. A detailed ablation study will be provided to validate the effectiveness of the proposed cross-view consistency loss. total loss is defined as
d
d
d
S,
Sile
net-
d
di
S
S
d: 10
Figure 2: Cross-view consistency loss. Monoc- ular depth or semantic segmentation maps are warped using dpst and dpst, and the consistency losses are measured using equation 3 and equa- tion 7 Dotted lines mean warping operator G(a; b). and solid lines denote the cross-view con- sistency losses. As summarized Table I, either spat or sf" used as supervision semantic seg- mentation.
Ls + Ld,e + Ls.c.
(11)
3.3 TRAINING DETAILS
While the pseudo depth labels d'et and dest gener- ated using pretrained stereo matching networks, are used to supervise the monocular depth estimation task, the semantic segmentation task requires using the ground truth segmentation maps. The Cityscapes dataset provide only the left ground truth segmenta- tion map se" and the KITTI dataset does not provide them. In our work, we hence generated the pseudo segmentation labels of these images by using seman- tic segmentation methods (Cheng et al. 2020 Zhu ett al., 2019).. Table [ summarizes the supervisions used for the two tasks.
Table 1: Supervision used in the KITTI and Cityscapes datasets.
Task
Input
KITTI
Cityscapes
sgt
sftt
Seg.
1,
shat
spar
1
dpst
dPss
Depth
1
dps:
dPgt
5

---

Under review as a conference paper at ICLR 2021
Table 2: Quantitative evaluation monocular depth estimation on Eigen split of KITTI dataset. Numbers in bold and underlined represent 1st and 2nd ranking, respectively. The methods used evaluation are (Garg et al. 2016), (Zhou et al. 2017). Monodepth (Godard et al.) 2017). Zhan
al. 2018), (Chen et al. 2019). Monodepth2 (Godard et al. 2019), Uncertainty (Poggi al. 2020
DepthHint (Watson et al. 2019). Guizilini et al. (020b), and (Choi et al. 2020).
(Lower better)
(Higher better)
Method
cap
Abs Rel
Sq Rel
RMSE
RMSE log
1.25
5 < 1.25²
< 1.25
Garg ct al.
0.152
1.226
5.849
0.246
0.784
0.921
0.967
Zhou ct al.
0.150
1.124
5.507
0.223
0.806
0,933
0.973
Monodepth
0.138
1.186
5.650
0.234
0.813
0,930
0,969
Zhan et al.
0.135
1.132
5.585
0.229
0.820
0.933
0.971
Chen ct al.
80m
0.118
0,905
5.096
0.211
0.839
0.945
0.977
Monodepth2
0.108
0.842
4.891
0.207
0,866
0,949
0.976
Uncertainty
0.107
0.811
4.796
0.200
0.866
0,952
0,978
DepthHint
0.102
0.762
4.602
0.189
0.880
0.960
0,981
Guizilini et al.
0.102
0.698
4.381
0.178
0.896
0.964
0,984
Choi ct al.
0.098
0.647
4.253
0.186
0.884
0,960
0.981
Ours
0.097
0.599
4.197
0.184
0.883
0.962
0.982
(a)
(b)
(d)
Figure 3: Qualitative evaluation of monocular depth estimation on Eigen split of KITTI dataset. (a) Input image, (b) Monodepth (Godard et al. 2017). (c) Monodepth2 (Godard et al. 2019), (d)
DepthHints (Watson et al. 2019), and (e) Ours.
EXPERIMENTAL RESULTS
4.1 DATASETS
We evaluated the performance on two popular datasets, KITTI (Geiger et al. 2012) and Cityscapes (Cordts et al. 2016). In KITTI, for fair comparison, we followed the common setup to use 22,600
images for training and the rest for evaluation. The Eigen split data (697 images) (Eigen al. 2014) was used for evaluating the monocular depth accuracy. Following existing MTT. methods (Chen al., 2019), the semantic segmentation accuracy was evaluated with 200 annotated data provided from KITTI benchmark. Cityscapes provides high resolution images of urban street scenes used for segmentation and depth estimation. 2,975 and 500 images were used for training and evaluation, respectively.
4.2 IMPLEMENTATION DETAILS AND EVALUATION METRIC
We first pretrained the monocular depth network E + and semantic segmentation network independently for 30 epochs using the Adam optimizer (Kingma & Bal 2015) with learning rate of 10-4 and momentum of 0.9. We then finetuned the whole network using the Adam optimizer with a learning rate of 10-5, reduced to 1/10 every 10 epochs, and a momentum of 0.9, after initializing with the pretrained weight parameters of the monocular depth network E + D. and semantic segmentation network D,. During training, we resized KITTI images resolution of [480, 192]. and cropped Cityscapes images [2048, 768] to exclude the front part of the car and resized to resolution of [256. 96]. The weights for the objective function are set to 850, a, = 2.5, Q..Ir 0.5, ax,i 1 ,0d,,r = I, Cx.Ir = 0.5, ax, = 1.5, a,,r 1.5. The performance evaluation was conducted by following the common practices: 1) mean absolute relative error (Abs Rel), mean relative squared error (Sq Rel), root mean square error (RMSE), root mean square error log (RMSE log) and accuracy under threshold 8 for monocular depth estimation. 2) intersection over union (IoU) and mean intersection over union (mloU) for semantic segmentation. Due to page limits, some results provided in appendix. Our code will be publicly available later.
D, for 20 epochs

---

Under review as a conference paper at ICLR 2021
Seprepatie
Figure 4: Qualitative results semantic segmentation prediction on KITTI dataset.
a
(b)
(c)
(d)
(f)
Figure 5:
Qualitative results of monoeular depth estimation on the Cityscapes dataset: (a) Input
image, (b) Ground truth, (c) Cross-stitch (Misra et al., 2016), (d) MTAN (Liu al., 2019), (c) Dense (Liu et al., 2019), and (f) Ours. Note that ground truth depth maps were obtained using SGM (Hirschmuller, 2008).
4.3 PERFORMANCE EVALUATION
KITTI In Table 2 we provide objective evaluation results the KITTI Eigen split (Eigen al. 2014). The proposed method produces very competitive results to state-of-the-arts monocular depth estimation approaches. Qualitative evaluation in 3 verifies that our method yields the results with sharper boundaries and better object delineation. These validate the effectiveness of the cross- view consistency based on the pseudo depth labels. In Figure 4 the proposed method produces satisfactory semantic segmentation results for the Cityscapes dataset, achieving mloU = 59.93.
Note that mloU in the MTI. approach of (Chen et al. 2019) is 39.13.
Cityscapes In Table 3 we compared re- sults the Cityscape dataset with recent multi-task learning approaches for monocu- lar depth estimation and semantic segmenta- tion tasks: 'Cross-stitch' (Misra et al. 2016) and 'MTAN' (Liu et al.) (019)). 'Split (deep)"
Table 3: Multi-task validation results for 7-class semantic segmentation and depth estimation on Cityscapes dataset.
Segmentation (Higher is better)
Depth
Method
"Split (wide)', and "Dense" were reproduced by using author-prov prohor-provided codes in 'MTAN' (Liu et al. 2011). Our method achieves improved quantitative results on both tasks. Figure 5 exhibits qualitative results the Cityscape dataset. As expected, depth and segmentation maps generated by our method are capable of preserving object boundaries and recover details better than the latest MTL methods (Misra et al. 2016 Liu et al. 2019).
(Lower is better)
mloU
loU
Abs
Sq
Split (deep)
56.24
88.13
0.644
0.259
Cross-stitch
59.77
89.70
0.610
0.251
Split (wide)
59.71
89.84
0.619
0.254
MTAN
51.08
89.77
0.669
0,250
Dense
60.01
89.81
0.598
0.242
Ours
64.76
91.90
0.542
0.228
4.4 A ABLATION STUDY
We conducted the ablation experiments to validate the effectiveness of the confidence map and cross- view consistency for the KITTI dataset in Table 4 and the Cityscapes dataset in Table 5 We first compared the performance with the method (b d,) based on the cross-view consistency using the estimated monocular depth map, e.g. ( G(s,; d:)), similar (Chen et al. 2019). Under the same setting, method (b dest tends to achieve higher mIoU than the method d,). Ad-
7

---

Under review as a conference paper at ICLR 2021
Table 4: Ablation study of our model on the KITTI dataset. 'Baseline' model is our network without the confidence and cross consistency loss.
Proposed components
Depth
Seg
Method
G(-:b)
(Lower is better)
(Higher is better)
Rel
RMSE
8
C2
Ladt
La
Abs Sq
raw log
1.25
1.25
1.25
mloU
Baseline
b deae
0.103 0.673
4.500/
0.871
0.957
0.980
56.90
d; de 0 d
0,101 0.665
4.510/ 0.191
0.873
0.958
0.981
58.26
0.103 0.674
4.537 0.196
0.869
0.956
0,980
58.44
0.104 / 0.678
4.579/ 0.196
0.867
0.955
0.979
59.00
b d
0.100 / 0.661
4,461 0.191
0.876
0.959
0.981
58.51
b 1
0.100 0.660
4.471
0.876
0.958
0.981
59.16
Ours
0 d
0.101 / 0.668
4.518/0.194
0.872
0.958
0.980
59.41
6 d
0.099 / 0.611
4.268 / 0.186
0.882
0.962
0,982
56,59
b d
0.096 / 0.612
4,285 185
0.884
0.962
0.982
58.91
b d
0.097 / 0.610
4.282 0.183
0.884
0.962
01983
59.78
b deee
0.096 0.616
4.287 0.184
0.884
0,962
0.982
60.21
b deac
0.100 0.613
4.213
0.878
0.961
0.982
56.79
0.097 0.599
4.197
0.883
0,962
0.982
59.93
Table 5: Ablation study of our model on the Cityscapes dataset. without the confidence and cross consistency loss.
'Baseline' model our network
Method
G((:b)
Proposed components
Depth (Lower is better)
Seg (Higher is better)
Lude
Latl Lih.
Abs
Sq
mloU
Pixel Ace
Baseline
b d.
0.584
0.246
63.01
91.24
b= d;
0.573
0.238
63.83
91.53
b d:
0.586
0.244
64.05
91.54
b d:
/
0.584
0.243
64.15
91.59
8 deg
0.572
0.242
63.85
91.30
d
0.559
0.237
64.32
91.68
Ours
d
0.572
0.242
64.36
91.37
6 du
0.565
0.236
63.17
91.30
b d
0.550
0.232
64.49
91.74
b doen.
0.552
0.234
64.56
91.80
dosc
0.553
0.235
64.68
91.88
b d. dR:
0.547
0.228
63.86
91.40
b
0.542
0.229
64.76
91.90
ditionally, while the method (b d,) often degenerates the monocular depth accuracy, method d =d does not suffer from such an issue, achieving the improved monocular depth accuracy. Such a performance gain become even more apparent for both tasks when using the confidence map. Note that it is infeasible to leverage the confidence map for the method (b d:) in which the esti- mated monocular depth map is constantly updated during the network training. When including the cross-view consistency loss La.e for monocular depth estimation, the additional performance gain was observed, validating its effectiveness on the monocular depth estimation. Though the segmen- tation accuracy (mloU) slightly worsen in some cases, it is relatively marginal. This may be due to our architecture where the two tasks share the encoder, and more advanced MTL architecture, e.g. using task-attention modules (Liu al. 2019). would lead performance improvement. We reserve this as future work.
(b
CONCLUSION
This paper has presented new MTL architecture designed for monocular depth estimation and semantic segmentation tasks. The cross-view consistency loss based on the pseudo depth labels, generated using pretrained stereo matching methods, was imposed on the prediction results of two views for resolving the mismatch problem. Intensive ablation study exhibited that it leads to sub- stantial performance gain in both tasks, especially achieving the best accuracy in the monocular depth estimation. Our task-specific losses can be used complementarily together with existing MTL architectures, e.g. based on task ssscifpcciiic attention modules (Liu et al. 2019). An intelligent com- bination with these approaches is expected to further improve the performance. Additionally, how to integrate recent architectures (Chen et al.) 2018; Taaiiaaa et al.) 2019) designed for semantic segmentation into the MTL network would be an interesting research direction.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
V. Badrinarayanan, A. Kendall, and R. Cipolla. SegNet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intel- ligence,
2495, 2017.
Jia-Ren Chang and Yong-Sheng Chen. Pyramid stereo matching network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Liang- -aighh Chen, George Papandreou. Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. CoRR, abs/1706.0555t 2017.
Liang- Chang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
Po-Yi Chen, Alexander H. Liu, Yen-Cheng Liu, and Yu Yu-Chiang Frank Wang. Towards scene un- derstanding: Unsupervised monocular depth estimation with semantic-aware representation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
Bowen Cheng, Maxwell D. Collins, Yukun Zhu, Ting Liu, Thomas S. Huang, Hartwig Adam, and Liang-Chich Chen. Panoptic-decplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Jaehoon Cho, Dongbo Min, Youngjung Kim, and Kwanghoon Sohn. A large RGB-D dataset for semi-supervised monocular depth estimation. CoRR, abs/1904. 10230, 2019.
Hyesong Choi, Hunsang Lee, Sunkyung Kim, Sunok Kim, Seungryong Kim, and Dongbo Min. Adaptive confidence thresholding for semi-supervised monocular depth estimation. CoRR, abs/2009.12840 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using multi-scale deep network. Advances in Neural Information Processing Systems (NIPS). 2014.
Ravi Garg, BG Vijay Kumar, Gustavo Carneiro, and Ian Reid. Unsupervised CNN for single view depth estimation: Geometry to the rescue. In European Conference on Computer Vision, 2016.
A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision bench- mark suite. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2012.
Clément Godard, Oisin Mac Aodha, and Gabriel J. Brostow. Unsupervised monocular depth estima- tion with left-right consistency. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow. Digging into self- supervised monocular depth estimation. IEEE International Conference on Computer Vision (ICCV), 2019.
Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon. 3D packing for self-supervised monocular depth estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020a.
Vitor Guizilini, Rui Hou, Jie Li, Rares Ambrus, and Adrien Gaidon. Semantically-guidedd repre- sentation learning for self-supervised monocular depth. In International Conference on Learning Representations, 2020b.
Caner Hazirbas, Lingni Ma, Csaba Domokos, and Daniel Cremers. Fusenet: Incorporating depth into semantic segmentation via dusion-based CNN architecture. In Asian Conference on Computer Vision (ACCV), pp. 213-228, 2016.

---

Under review as a conference paper at ICLR 2021
Heiko Hirschmüller. Stereo processing by semiglobal matching and mutual information. 1EEE Trans. Pattern Anal. Mach. Intell., 30(2):328-333 341, 2008.
Ankit Jha, Awanish Kumar, Biplab Banerjee, and Subhasis Chaudhuri. AdaMT-Net: An adaptive weight learning based multi-task learning model for scene understanding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Varun Ravi Kumar, Senthil Kumar Yogamani, Markus Bach, Christian Witt, Stefan Milz, and Patrick Mäder. UntectDepthNet: Self-supervised monocular depth estimation using a generic framework for handling common camera distortion models. CoRR, abs/2007 00666, 2020.
Shikun Liu. Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Chenxu Luo, Zhenheng Yang. P. Wang, Y. Wang, W. Xu, R. Nevatia, and A. Yuille. Every pixel counts ++: Joint learning of geometry and motion with 3D holistic understanding. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 42:2624 26411,266,,, 2020.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Jiahao Pang, Wenxiu Sun, Jimmy SJ Ren, Chengxi Yang, and Qiong Yan. Cascade residual learning: A two-stage convolutional neural network for stereo matching. In ICCV Workshop Geometry Meets Deep Learning, Oct 2017.
Matteo Poggi and Stefano Mattoccia. Learning from scratch confidence measure. In British Machine Vision Conference (BMVC), 2016.
Matteo Poggi, Filippo Aleotti, Fabio Tosi, and Stefano Mattoccia. On the uncertainty of self- supervised monocular depth estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015.
Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik G. Learned-Miller, and Jan Kautz. Pixel- adaptive convolutional neural networks. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2019, 2019.
Towaki Takikawa, David Acuna, Varun Jampani, and Sanja Fidler. Gated-SCNN: Gated shape enns for semantic segmentation. IEEE International Conference on Computer Vision (ICCV), 2019.
Alessio Tonioni, Matteo Poggi, Stefano Mattoccia, and Luigi di Stefano. Unsupervised domain adaptation for depth prediction from images. IEEE Trans. Pattern Anal. Mach. Intell., 42(10): 2396-2409, 2020.
J. Uhrig, N. Schneider, L. Schneider, U. Franke, T. Brox, and A. Geiger. Sparsity invariant CNNs. In International Conference on Vision (3DV), pp. 11-20, 2017.
Lijun Wang. Jianming Zhang, Oliver Wang, Zhe Lin, and Huchuan Lu. SDC-Depth: Semantic divide-and and-uuuuer network for monocular depth estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 338-557,20 2020.
Jamie Watson, Michael Firman, Gabriel J. Brostow, and Daniyar Turmukhambetov. Self-supervised monocular depth hints. In IEEE International Conference Computer Vision (ICCV), 2019.
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic seg- mentation. CoRR, abs/1909.1 1065, 2019.
10

---

Under review as a conference paper at ICLR 2021
Amir R. Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Amir Roshan Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas J. Guibas, Robust learning through cross-task consistency. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2020.
Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian D. Reid. Unsupervised learning of monocular depth estimation and visual odometry with deep fea- ture reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Zhenyu Zhang, Zhen Cui, Chunyan Xu, Zequn Jie, Xiang Li, and Jian Yang. Joint task-recursive learning for semantic segmentation and depth estimation. In European Conference on Computer Vision (ECCV), pp. 238 255, 2018.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and ego-motion from video. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Yi Zhu, Karan Sapra, Fitsum A. Reda, Kevin J. Shih, Shawn Newsam, Andrew Tao, and Bryan Catanzaro. Improving semantic segmentation via video propagation and label relaxation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
11

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 MORE COMPREHENSIVE EVALUATION RESULTS FOR KITTI
We provide more comprehensive results for KITTI dataset. Fioure the qualitative evaluation with existing monocular depth estimation methods on Eigen Split of the KITTI dataset. Figure 7 shows the semantic segmentation prediction results the KITTI dataset. We also evaluated the performance with the improved ground truth depth maps made available at (Uhrig et al.) (017) for the KITTI dataset in Table Our approach is state-of- the-an art monocular depth estimation compared to existing methods.
(a)
(b)
(c)
Figure 6: Qualitative evaluation of monoeular depth estimation on KITTI dataset. (a) Input image, (b) Monodepth (Godard et al. 2017), (c) Monodepth2 (Godard et al. 2019), (d) DepthHints (Watson ct al. 2019), and (e) Ours.
Table 6: Quantitative results of monocular depth estimation using the improved ground truth depth maps made available at (Uhrig et al., 2017) for the KITTI dataset. Numbers bold and underlined represent 1st and 2nd ranking, respectively. The methods used in evaluation are EPC++ (Luo al. 2020 Monodepth2 (Goddad et al. 2019), Uncertainty (Poggi et al., 2020), Packnet-Sfi (Guizilini
al. 2020a), UnReciDeepthNet [Kumar et al. 2020), and Choi al. (Choi et al. 2020
Lower is better
Higher is better 1.25²
Method
Abs Rel
Sq Rel
RMSE
RMSE log
8 < 1.25
8 << 1.253
EPC++
0.120
0.789
4.755
0.177
0,856
0.961
0.987
Monodepth2
0.090
0.545
3.942
0.137
0.914
0.981
0.994
Uncertainty (Boot+Iar Log)
0.085
0.511
3.777
0.137
0.913
0.980
0.994
Uncertainty Selot+Self)
0.085
0.510
3.792
0.135
0.914
0.981
0.994
Uncertainty Logg
0.084
0.529
3.833
0.138
0.914
0,980
0.994
Uncertainty (Snap+Self)
0.086
0,532
3.858
0.138
0.912
0.980
0,994
Packnet-SfM
0.078
0.420
3,485
0.121
0.931
0.986
0,978
UnRect UnetctDepthNet
0.081
0.414
3.412
0.117
0.926
0.987
0,996
Choi et al.
0.078
0.357
3,203
0.120
0.930
0.987
0,996
Ours
0.076
0.322
3.094
0.117
0.933
0,988
0.997
Input
Segmensation GT
Our prediction
Figure 7: Qualitative results of semantic segmentation prediction on the KITTI dataset.
12

---

Under review as a conference paper at ICLR 2021
A.2 MORE COMPREHENSIVE EVALUATION RESULTS FOR CITYSCAPES
We provide more comprehensive results for the Cityscapes dataset. Figure 8 and 9 show the qualita- tive evaluation results with existing MTL methods (Misra et al. 2016 Liu al. 2019) for monocular depth estimation and semantic segmentation on the Cityscapes dataset. These results also support the effectiveness of our method.
(a)
(d)
Figure 8: Qualitative results of monocular depth estimation on the Cityscapes dataset: (a) Input im- age, (b) Ground truth depth map obtained using SGM (Hirschmüller 2008), (c) Cross-stitch (Misra ct al. 2016). (d) MTAN (Liu et al. 2019), (e) Dense (Liu et al. 2019 and (f) Ours.
(a)
(b)
(d)
(c)
Figure 9: Qualitative results of semantic segmentation results on the Cityscapes dataset: (a) Input
image, (b) Ground truth segmentation map. (c) MTAN (Liu al. 2019 (d) Cross-stitch (Misral
et al. 2016. (e) Dense (Liu et al. 2019). and (f) Ours.
A.3
QUALITATIVE EVALUATION FOR ABLATION STUDY
Figure and and shows the qualitative evaluation for ablation study the KITTI and Cityscape datasets, respectively. It was found that our final results are much improved compared to those of the 'Baseline' model, validating the effectiveness of the confidence map and cross- view consistency loss.
13

---

Under review as a conference paper at ICLR 2021
(a)
(b)
(c)
(d)
ff
g
Figure 10: Qualitative evaluation for ablation study on the KITTI dataset: (a) Depth input images, (b) Depth map with 'Baseline' model, (c) Our final depth map, (d) Segmentation input images, (e) Ground truth segmentation map (f) Segmentation map with 'Baseline' model, (g) Our final
segmentation result. 'Baseline' model is our network without the confidence and cross consistency loss.
(c)
(c)
ff
g
(h)
Figure 11: Qualitative evaluation for ablation study on the Cityscape dataset: (a) Depth input im- ages, (b) Ground truth depth map, (c) Depth map with 'Baseline' model, (d) Our final depth map, (e) Segmentation input images, (f) Ground truth segmentation map, (g) Segmentation map with 'Baseline' model, (h) Our final segmentation result. 'Baseline' model our network without the confidence and cross consistency loss.
14


Reviews:
[{"review_id": "zkWSLA5i8gP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose to use depth pseudo ground truth (generated with a pretrained stereo network) as augmented information to help a joint prediction network for depth and segment estimation. \n\nPros:\nMulti task learning is an important direction to explore, and left-right consistency has shown to be very useful in depth estimation Godard et.al 2017. The extension using similiar idea to depth and semantic is reasonable, and experiments verify the effectiveness of proposed strategies. \n\nCons: \n1) Scope: \nIt seems the paper works specific on the left-right warping consistency of semantic label and depth, while the major scop told in the title and introduction is about pseudo label for general multiple task learning, which is byfar not shown in the worked experiments. It needs to be adjusted.  \n\n\n2) Method:\nThe major methodology is using obtain consistency losses by warping depth and semantic with respect to stereo output. The warped loss containing 6 terms each through enumeration, are all of them useful ? Is there a lot of redundency, what happened if droping half of it. The ablation shows using concistency is useful, while the usefulness of each term and how balance between these losses has not been proven. \n\n\n3) Experiments\nComparing to other SoTA algorithms, it seems for depth, the results are comparable to many existing algorithms, and for semantic it is hard to compare against other SoTA semantic algorithms such as HRNet etc.. In my opinion, MTL has two benefits either differet tasks can help the output results, another is unifying tasks into single network for more efficient inference. It might be better to also compare about the running speed and Flops for performing multiple tasks to better support the idea. \n\n4) Writting\nOverall, it is easy to follow, however the figures are too small making it hard to diagnose the difference between multiple predictions. \n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "vd8T39GylIi", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a framework to learn depth prediction and semantic segmantation jointly; the key idea lies in making use of the pseudo depth label from stereo to provide supervision and as a means to enforce cycle consistency between the left and right views of the stereo. \n\nReasons for scores: overall the paper is rather incremental and the idea is neither novel nor significant in my opiniont. I do not see interesting or deep insight from the paper towards the depth and semantic segmantation tasks.  \n\nPros:\n+ First, the paper is clearly written and easy to follow. The proposed framework is pretty straightforward. \n+ The idea of joint learning depth and semantic segmentation is good considering their tightly coupled nature. \n+ The use of cross-view consistency as a constraint is good.\n\nCons:\n- Overall, the paper does not have much novelty in my opinion. Joint learning of depth and semantic segmentation is clearly not new, and the paper does not provide new or particular insight towards this combined learning.\n- The use of pseudo label itself is nowadays quite common in the vision community.  And, the pseudo labels are used in the paper in a pretty trivial way in my opinion. \n-  The cross-view consistency across two views in a stereo setup is not new neither. It has been intensively used in the monocular depth estimation. In addition, this constraint is applicable to any individual task and does not seem to fit into the multi-task learning context, which is the main focus of this paper. I would expect specific insights in making use of pesudo labels towards solving the depth and semantics predictions; otherwise, any other tasks such as moving objects segmentation \n- The current title is too general, so much so that the main arguments made by the paper are not reflected in the title; I believe that the left-right consistency brought about by the pseudo ground truth depth is the main claim of the paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "R9Iqc8HgmuR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a joint learning strategy for simultaneous semantic segmentation and monocular depth estimation. The main idea is to exploit stereo pairs in training and introduce pseudo-depth label estimated from pre-trained stereo-matching networks. Given the pseudo-depth with confidence estimation, the method proposes a cross-view consistency loss for both depth and semantic predictions, which augments the standard segmentation loss. The proposed method is evaluated on KITTI and Cityscapes datasets with comparisons to prior work and ablative study.   \n\nStrengths:\n- The proposed cross-view loss on semantic segmentation seems interesting and effective on two benchmarks, which improves the segmentation performance. \n- The overall method achieves competitive performance on semantic segmentation and monocular depth estimation on the KITTI and Cityscapes. \n\nConcerns:\n- The contribution of this work to the multi-task learning is a bit overclaimed. The targeted problem of the paper is solely on joint semantic segmentation and monocular depth estimation. Based on the model and loss design, it is non-trivial to extend them to other scene understanding tasks. \n\n- The problem setting in this work, which requires stereo image pair for learning network, is different from the prior work  (e.g., Liu et al 2019). The proposed method also uses a pre-trained stereo-matching networks and confidence estimation network, which essentially included additional prior information/training data. Therefore, it is not surprising to see the performance improvement over the prior work. \n\n- While the proposed cross-view loss improves the segmentation, the overall design is quite complicated. There are many hyper-parameters in the loss functions, and it is unclear how their values would generalize to other datasets that are not road scenes. Moreover, based on the ablative study, the improvement over the noisy depth setting is marginal (Table 4 and 5). Also, it is unclear whether all those terms make significant contributes to the performance improvements, and sometimes it even hurts the performance. \n\n- The experimental evaluation is a bit lacking in the following aspects. \n  + This work only uses two road-scene datasets for evaluation, but those two datasets are quite similar to each other, and hence do not have sufficient diversity. The other work typically also use NYU-v2, which is an indoor dataset. Can the author also report their method's performance on NYU-v2?\n\n  + The evaluation on the Cityscapes dataset seems unconvincing due to two issues: First, the depth performance in Table 3 seems very different from the prior literature, and in particular, the Abs values are much worse than the SOTA results. Secondly, it lacks comparisons with Jha et al. 2020, which achieves better performance than the results shown in Table 3. \n\n  + The improvement from the proposed loss seems very marginal in the ablative study. Different combinations of proposed components typically give minor or mixed improvement on segmentation or depth estimation. It is unclear how effective of the confidence weighting or using multiple consistency constraints. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the paper overclaims its contributions both in the main text and in the title, and given also the limited novelty  and scope it is not suggested for publication.
Decision: Reject