Title: UNSUPERVISED MONOCULAR DEPTH ESTIMATION WITH CLEAR BOUNDARIES

Abstract: Unsupervised monocular depth estimation has made great progress after deep learning is involved. Training with binocular stereo images is considered as a good option as the data can be easily obtained. However, the depth or disparity prediction results show poor performance for the object boundaries. The main reason is related to the handling of occlusion areas during the training. In this paper, we propose a novel method to overcome this issue. Exploiting disparity maps property, we generate an occlusion mask to block the back-propagation of the occlusion areas during image warping. We also design new networks with flipped stereo images to induce the networks to learn occluded boundaries. It shows that our method achieves clearer boundaries and better evaluation results on KITTI driving dataset and Virtual KITTI dataset.

1. INTRODUCTION:

Monocular depth estimation becomes an active research topic as deep learning is applied in various computer vision tasks. It has many applications, from navigation through to scene understanding. A single traditional camera can be a cheaper alternative to the expensive LIDAR sensor for automotive cars if accurate estimation can be achieved. Meanwhile, single camera simplifies the design of depth estimation solution which can be adopted quite widely at a low cost. shows clear boundaries with less artifacts and higher accuracy One straight-forward way to train deep depth estimation models is to use ground truth depth images as the supervision signals Eigen et al. (2014) . However, supervised deep learning method is eager for massive data with ground truth. Collecting large datasets with ground truth depth in varied real scenarios is challenge and expensive. Instead, training using stereo images without depth label is an alternative option. Godard et al. (2017) proposed a method to exploit the left-right consistency of stereo images to tackle the monocular depth estimation, which achieved quite promising results. However, the depth predicted by their method has blurred boundaries. The issue is mainly due to the occlusions during the image warping. Though it can be alleviated in some extent with proper post processing, the fundamental problem is not well addressed.

In this paper, we propose a new method to overcome the blurred boundaries when using stereo pairs to train the monocular depth model. An example is illustrated in Fig. 1 . During the image warping, we generate an occlusion mask using the disparity map to block the inappropriate back-propagation gradients for occlusion areas. However, the mask only cannot guarantee clear boundaries as there is no constrain for the masked areas. Then we design new networks to fully exploit the information of stereo images. With flipped stereo pairs, the network is induced to learn clear boundaries for occlusion areas. Our method provides a solution to the fundamental learning difficulty of occluded areas introduced by image warping in depth estimation. Empirical evaluation on KITTI driving dataset (Geiger et al. (2012) ) and Virtual KITTI dataset (Gaidon et al. (2016) ) demonstrates the effectiveness of our approach. Moreover, we find the depth label of KITTI 2015 is usually very sparse near the object boundaries, which is not very sensitive to evaluate the clearness of boundaries.

2. RELATED WORK:

Large amounts of multi-view based approaches have been proposed such as stereo matching (Scharstein & Szeliski (2002) ), difference view point (Furukawa et al. (2015) ) or temporal sequence (Ranftl et al. (2016) ). Here we briefly review work based on single view depth estimation which is usually harder since reasoning depth from monocular colored image is an ill-posed problem.

2.1. SUPERVISED MONOCULAR DEPTH ESTIMATION:

The most intuition way is treating monocular depth estimation problem as a supervised problem by taking RGB images as inputs and Lidar depth points as ground truth. Saxena et al. (2009) proposed a method known as Make3d which breaks the image into homogeneous patches. For each small homogeneous patch, Saxena et al used a Markov Random Field (MRF) to infer a set of plane parameters that capture both the 3D location and 3D orientation. However, this approach has a hard time capturing thin structures since the predictions are made locally. Eigen et al. (2014) first exploited CNN in a coarse to fine manner. Liu et al. (2016) Liu et al. (2015) proposed a network jointly explore the capacity of deep CNN and continuous conditional random field (CRF). Ladicky et al. (2014) incorporated semantic segmentations in to single-view depth estimation task since they are closely tied to the property of perspective geometry. Laina et al. (2016) proposed a residual network with up-sampling module using fully convolutional architecture. Also, reverse Huber loss was introduced. However, large amount of high-quality labelled data is needed, which is hard to require in practice.

2.2. SEMI-SUPERVISED/UNSUPERVISED BASED METHODS:

To overcome the lack of high quality labelled data, several semi-supervised and fully unsupervised methods have been proposed. Flynn et al. (2016) first proposed a view synthesis based method called DeepStereo which generates new view image from nearby image. Xie et al. (2016) (Deep3D) proposed a method which generates the right image through probability distribution over all the possible disparities for each pixel. Garg et al. (2016) first proposed a warp based method by aligning reconstructed image with the ground truth left image as described in 3.1. However, their loss is not fully differentiable. Godard et al. (2017) improve this methods by introducing a novel loss. Repala & Dubey (2018) extended the network into two separate channel with 6 or 12 losses which improves the result.

Based on Garg et al, Kuznietsov et al. (2017) proposed a semi-supervised methods that exploited both the sparse Lidar points as supervision and stereo pairs as unsupervion signals. The semisupervised method was further improved by Luo et al. (2018) , they decoupled the monocular depth prediction problem into two procedure, a view synthesis procedure followed by stereo matching.

Recently, several work using only monocular temporal sequence comes out which enables more training data such as video sequence on YouTube. Zhou et al. (2017) proposed a network that predicts depth and camera pose separately. Using the predicted depth and camera pose, relative temporal image can be reconstructed by image warping with which final loss can be constructed. Mahjourian et al. (2018) performed a novel 3D loss that enforces consistency of the estimated 3D point clouds and ego-motion across consecutive frames and combined it with 2D photometric loss. Wang et al. (2018) proposed a differentiable implementation of Direct Visual Odometry (DVO) and a novel depth normalization strategy. However, all the temporal sequence based training meet the same problem of object motion. This problem can be alleviated by including stereo pairs during training known as trinocular training (Zhan et al. (2018) Poggi et al. (2018) Godard et al. (2018) ). However, all these warp based methods have the difficulty of learning occluded area which would infect the result.

3. METHODS:

Godard et al achieves state of art result of unsupervised monocular depth estimation with only stereo images. Follow Godard et al, we proposed monocular depth estimation network with novel mask methods which can be trained end-to-end and without ground-truth label. Our method is superior to Godard et al in result quality with clearer boundaries, especially on dense evaluation dataset such as virtual-KITTI (Gaidon et al. (2016) ).

3.1. SELF-SUPERVISED DEPTH ESTIMATION FROM STEREO PAIRS:

In general, our goal is to learn a network that can predict a pixel-wise dense depth map from single colored image(d l = d(I l )). However, all supervised methods have a hard time with acquiring large dense labelled data in real scenarios. Thus, several unsupervised methods have been proposed to overcome the obstacle. Among these methods, training by image reconstruction using rectified stereo pairs became more and more popular currently due to its high accuracy and easy accessibility of training data. (Garg et al. ( 2016 The network was further improved by Godard et al. (2017) by introducing left-right consistency loss and refined encoder-decoder network. Given the input left image, the network predicts both the left and right disparities simultaneously which enables constructing the left-right consistency. This consistency restriction leads to more accurate result and less artifacts. Also, fully differentiable backward bilinear sampling was used to reconstruct the left image which makes the model easier to optimize. With better network architecture and better loss, Godard et al achieved the state of art result of unsupervised monodepth estimation only with rectified stereo pairs, and even outperforming supervised methods. Their network architectures are shown in Fig. 2 .

However, there still are unsatisfactory artifacts at the occlusion boundaries showing blurred ramps on the left side of image and of the occluders. Even with the post-processing step which weighted sums the flipped disparity of the flipped input image and the disparity of the input image, the blurred ramps are still visible near the objects, especially for those near the cameras with higher disparities as illustrated in Fig. 1 .

3.2. WARPING MASK:

Though common backward warping using bilinear sampler is fully differentiable which enables us to train the model end to end, some undesirable duplicates and artifacts are introduced during the warping process because of occlusions according to Lu et al. (2018) . In Fig. 3 , we use SYTHIA dataset (Ros et al. (2016) ), a synthesized virtual driving dataset to illustrate. Though ground truth disparities are used for warping, there still exists obvious duplicates and even a huge black region on the left of the reconstructed image( Ĩl ) because of the occlusion. If those inevitable artifacts and duplicates are back propagated during the training process, unwanted high losses will be introduced forcing the network learn to blur in those regions, as the blurriness(disparity ramps) in the occluded regions will make the reconstructed image show stretched patterns(Fig 7 ) which are more similar to original ground truth image compared to duplicates and large black regions(Fig. 3 ). In order to block the back propagation process of those warping induced artifacts, we designed an algorithm which can mask the occlusion region automatically. In backward bilinear sampling process, a disparity map was used to sample from the source image. The intuition here is that, if any pixel in the source image has never been sampled during the bilinear sampling process, this pixel should only be visible in the source image and should be masked when reconstructing this source image later on. Thus, the mask methods takes, say, left disparity as input while generating the mask of reconstructed right image and vice versa. The pseudo code is shown in Algorithm 1 and the mask schematic diagram is shown in Fig 4 . However, the mask method alone cannot guarantee the clearness since no constrain is added in masked region. Specifically, though the masks block the back propagation of duplicates and artifacts induced by warping, they also block the further learning process of those regions. Once the disparities start to blur, hardly can we correct the network back to clearness. To solve this problem, we refined the network architecture and introduced a flip-over training scheme.

3.3. SOLVING THE PROBLEM OF BLURRINESS:

Though the mask blocks the process of further learning of the blurred regions, we find that the blurred side are definite which can be exploited to reactive the learning process. For example, when the network is only trained on left images and takes corresponding right images as ground truth, the disparity ramps (blurred regions) will only appear on the left side of the occluders. So, if we randomly flipped the input images horizontally, the disparity ramps will still appear on the left side. When flipped the output disparities back for warping, the blurred regions will appear on the right side where no mask is added. Examples are shown in the last column of Fig 7 . Thus, those blurred regions will make distortions when reconstructing images, and back propagate despite the Algorithm 1: mask algorithm for occluded regions Input: disparity of left image d l ,disparity of right image d r ,shape of input image(h,w) Output: mask of reconstructed image mask r and mask r (Boolean array with the same height and width with the images, where Trues are masked region ) mask r ← T rues(h, w); mask l ← T rues(h, w); for x,y in grid(h,w)

do if 0 ≤ x -d l (x) ≤ w then mask r 1 (f loor(x -d l (x)), y) = F alse; mask r 2 (cell(x -d l (x)), y) = F alse; mask r = mask r 1 ∨ mask r 2 end if 0 ≤ x + d r (x) ≤ w then mask l 1 (f loor(x + d r (x)), y) = F alse; mask l 2 (cell(x + d r (x)), y) = F alse; mask l = mask l 1 ∨ mask l 2

end end return mask l , mask r mask. Also, because the flip-over scheme is performed randomly, any blurriness on the definite side will receive punishment on average which restrict the prediction to be clear. It is worth to mention that flip-over scheme will not affect masking process and we still use masks to block the back propagation of duplicates. The flip-over schematic diagram is shown in Fig. 4 . However, the predicted disparities of the right branch won't make any sense if we take the flipped left image as input and are totally mismatched with the ground truths as is shown in Fig 4 . As a result, we delete the predicting branch of the right disparity and add another encoder-decoder network which takes right images as input and predicts right disparities as is shown in Fig 2 . This doubled encoder-decoder network enables us to preform left-right consistency loss at the cost of doubling the training parameters. However, it won't slow down the test speed since only one branch of encoder-decoder network is used when testing. We also tried another network architecture with shared the encoder-decoder network which achieves comparable result as non-shared network while halves the training time. More details can be found in 6.1

3.4. TRAINING LOSS:

We use similar training loss as Godard et al. (2017) . The losses are performed on four different scale and are finally summed as the total loss. C = 4 s=1 C s . For each scale, three different losses including appearance match loss(C ap ), disparity smoothness loss(C ds ) and LR consistency loss(C lr ) are performs as follows.

EQUATION

Intuitively, appearance matching loss measures photometric error between the reconstructed image( Ĩl ij )) and the ground truth image(I l ij ), which is defined as the weighted sum of L1 and SSIM shown as follows,

EQUATION

Disparity smoothness loss aims to force the smoothness of predicted disparities through

C l ds = 1 N i,j ∂ x d l ij e -∂xI l ij + ∂ y d l ij e -∂yI l ij .

(3) Finally, L1 penalty is added to constrict the left-right consistency, which further reduce the artifacts,

C l lr = 1 N i,j d l ij -d r ij+d l ij .

(4)

3.5. IMPLEMENT DETAILS:

Our network is based on Godard et al. (2017) and is implemented in Tensorflow (Abadi et al.) . The network contains encoder network based on VGG16 or Resnet50 and decoder network with 7 upconvolutional layers, 7 merging layers and 4 disparity prediction layers. With 7 skip connections, the network can better handles features at different scales. Different from Godard et al, we modified the channel number of disparity prediction layers to predict only one disparity instead of two.

Also, we tuned the default hyper parameters α and α ds . The rest are the same with Godard et al with α ap = 1, α lr = 1. The learning rate λ remains 10 -4 for the first 30 epoch and halves every 10 epoch when trained for 50 epoch. We also tried batch norm but might lead to unstable result. Data augmentation is performed on the fly similar as Godard et al including flipping and color shifting.

4. EXPERIMENT:

We trained our model on rectified stereo image pairs in KITTI and Cityscapes dataset and evaluated mainly on KITTI split (Geiger et al. (2012) ), Eigen split (Eigen et al. (2014) ) and virtual-KITTI datasets (Gaidon et al. (2016) ). Also, we find the problem of KITTI 2015 evaluation such as sparsity and man-made defects which infects our evaluation result. When testing on dense datasets like virtual-KITTI, the superiority becomes more obvious. high quality disparity image issued by official KITTI dataset. It is worth mentioned that though these disparity images are of better quality than projected velodyne laser point and with CAD models inserted in cars, most part of ground truth are extremely sparse, especially in terms of occluded regions. As the red boxes in the Fig. 5 shown, there is some blank space on the left side of image and occluded regions which is not involved in the evaluation. Thus, those blank space just covers the shortcomings the disparity ramps on the occluded regions. As a result, our result shows less superiority over Godard et al on KITTI stereo 2015 dataset. 

As shown in:



5. CONCLUSION:

In this work, we present an occlusion mask and filp-over training scheme to enable effective learning of object boundaries when using image warping. With our new network, our model achieves state of art result using only stereo images. Moreover, as warping based image reconstruction is commonly used in depth estimation problem, our method provides a solution to the fundamental difficulty of occluded areas introduced by image warping.

In the future, our method can be incorporated with more accurate network trained on trinocular data (temporal Stereo sequence) such as Zhan et al. (2018) , Poggi et al. (2018) and Godard et al. (2018) , which would further boost the accuracy.

regions are no longer aligned with the masks which enables further learning process. As the training process goes on, the boundaries will become clearer as shown in Fig. 1 . We also use schematic diagram to illustrate in Fig. 4 Figure 7 : Example of blurred results and flip-over scheme.

:



Figures and Tables:

Figure 1: Blurred boundaries of predicted disparity by Godard et al. (2017) and our result. (a) input image (b) predicted left disparity (c) post-processed disparity(d) our result. To eliminate the artifacts and blurred patterns, Godard et al weighted sum the disparity of the input image and flipped disparity of the flipped input image at the cost of doubling the computation. However, there still exists blurred patterns in the post-processed disparity. Contrary to Godard et al. as shown in (b)(c), our result(d) shows clear boundaries with less artifacts and higher accuracy

Figure 2: Network architecture. From left to Right: Naive unsupervised network proposed by Garg et al. (2016), Left-right consistency network proposed by Godard et al, Our refined network without shared parameters and our refined network with shared parameters First proposed by Garg et al. (2016), the monodepth estimation network takes left image (I l ) as input and outputs the disparity aligned with the right image (d r ) which can be used to sample from the left image (I l ) to reconstruct the right image ( Ĩr ) during training. Thus, image reconstruction loss can be constructed between the reconstructed right image( Ĩr ) and original right image(I r ). When testing, only one colored image are required, the predicted disparity can be converted to depth simply using depth = b * f /disparity, where b and f are given baseline and camera focal length respectively. It is worth to mention that disparities (d l and d r ) are a scalar per pixel as the images are rectified.

Figure 3: Problem of warping and corresponding mask. (a)left image I l (b)right image I r (c)disparity aligned with left image d l (d)reconstructed left image Ĩl (e)reconstructed left image after being masked. We used ground truth disparity of left image d l (c) to warping the right image I r (b) by bilinear sampling. However, duplicates and artifacts are obvious in reconstructed left image ( Ĩl )(d) compared to the ground truth left image (a). We used a warping mask which is generated automatically from disparities to block the back-propagation of those artifacts. The final output is shown in (e), where the white regions are masked.

Figure 4: Schematic diagram. The orange squares refer to an object such as pedestrians or vehicles.The blue and black squares refer to blurred disparity region and masked region respectively. Left: Naive masking process. Warping mask(black) is overlap with blurred region(blue). As a result, masks will block the learning of those region. Right: Flip-over training scheme. We flipped the left images(f (I l )) as input and flipped the output disparities(f (d(f (I l )))) back to reconstruct the image( Ĩl ). Different from the diagram shown in left, the blurred region will switch to the opposite side to avoid the mask. As a result, losses will be introduced leading to clear boundaries. We can also observed that the right branch (flipped predicted right disparity of flipped left image(f (d(I r )))) is totally mismatch with the ground truth right image, so we delete this branch and add another encoder-decoder branch as shown in Fig.2.

KITTI SPLIT For comparison, we use the same split as Godard et al. (2017) which 29000 out of 42382 rectified stereo image pairs with size of 1242*375 are used for training. The evaluation is performed on 200

Figure 5: The sparsity problem of KITTI 2015 evaluation split. Vacancy are visible on the left of the image and near the boundaries of cars and trees, which would have an impact on the evaluation result.

Table1, we use the metrics fromEigen et al. (2014) and D1-all metrics from KITTI. Our model achieves comparable result with Godard et al when both trained on KITTI dataset only for 50 epoch, and superior result when both trained longer for 100 epoch. Evaluation result on KITTI 2015 stereo dataset(Geiger et al. (2012)). K means the model is only trained on KITTI dataset, while CS + K means the model is trained on Cityscapes dataset and then finetune on KITTI dataset. Also, pp means post processing step, and more details about post processing can be found in 6.3. For a fair comparison, we train the network proposed by Godard et al and our non-shared network both to 50 and 100 epoch, and find that our network has larger improvement and better performance thanGodard et al. (2017) when trained longer to 100 epoch. We guess that our network requires longer time to converge. We choose different hyperparameters from Godard et al for our network. Though it is unfair to evaluation on sparse KITTI dataset, our result still outperforms that of Godard et al.4.2 EIGEN SPLITSimilar to Godard et al, we use the test split of 697 images proposed byEigen et al. (2014). Each image contains 3D points captured by velodyne laser which was used to generate ground truth depth. We keep the same 22600 stereo pairs for training. According to Godard et al, all evaluation results are done under the Garg crop(Garg et al. (2016)) except for Eigen results for a fair comparison. We also present the uncropped result which our models' superiority become more obvious, because it will crop the disparity ramps on the left side for evaluation which boosting Godard et al's result. Also, ground truths are captured by velodyne Lidar thus rather sparse which would reduce our superiority over Godard et al. The evaluation result is in Table 24.3 VIRTUAL-KITTITo prevent sparsity induced inaccurate result, we evaluate models on the Virtual-KITTI dataset(Geiger et al. (2012)). Virtual-KITTI dataset contains 50 monocular videos generated from five different virtual worlds in urban settings under different weather conditions with corresponding pixel-level ground truth depth. With the same resolution, scenes and camera parameters as KITTI 2015, Virtual-KITTI dataset can be implement naively when testing. Since KITTI 2015 dataset we used for training does not cover those weathers, only 2126 labbeled images without weather condi-Method Supervised Dataset Abs Rel Sq Rel RMSE RMSE log δ < 1.25 δ < 1.25 2 δ < 1.25 3

Table content: <table><tr><td>Godard 50 epoch</td><td>vgg</td><td>K</td><td>0.124</td><td>1.388</td><td>6.125</td><td>0.217</td><td>30.272</td><td>0.841</td><td>0.936</td><td>0.975</td><td/></tr><tr><td>Godard 100 epoch</td><td>vgg</td><td>K</td><td>0.124</td><td>1.378</td><td>6.146</td><td>0.216</td><td>31.522</td><td>0.837</td><td>0.936</td><td>0.975</td><td>Lower is better</td></tr><tr><td>Godard resnet pp</td><td>resnet</td><td>CS + K</td><td>0.097</td><td>0.896</td><td>5.093</td><td>0.176</td><td>23.811</td><td>0.879</td><td>0.962</td><td>0.986</td><td/></tr><tr><td>Ours non-shared 50epoch</td><td>vgg</td><td>K</td><td>0.124</td><td>1.415</td><td>6.126</td><td>0.215</td><td>30.988</td><td>0.842</td><td>0.937</td><td>0.975</td><td>Higher is better</td></tr><tr><td>Ours non-shared 100epoch</td><td>vgg</td><td>K</td><td>0.121</td><td>1.228</td><td>5.977</td><td>0.209</td><td>31.10</td><td>0.846</td><td>0.941</td><td>0.977</td><td/></tr><tr><td>Ours non-shared resnet pp</td><td>resnet</td><td>CS + K</td><td>0.097</td><td>0.960</td><td>5.030</td><td>0.170</td><td>22.585</td><td>0.884</td><td>0.963</td><td>0.985</td><td/></tr></table>

Result on Eigen splitEigen et al. (2014). K is KITTI and CS is cityscapes dataset for training. We use the evaluation result provided inGodard et al. (2017). For a fair comparison, we use the crop the same asGarg et al. (2016) except forEigen et al. (2014) and apply the same hyper-parameters as Godard et al on our model. Besides, we set the maximun evaluation depth to 50 meters(cap 50m) in the second row which is the same asGarg et al. (2016) , while others remain 80 meters. Also, we compared uncropped result with Godard et al, on which the superiority become more obvious. tion are used for evaluation. The evaluation result is shown in Table3. For visual result, please see our supplementary 6.2.

Table content: <table><tr><td>Method</td><td colspan="8">Dataset Abs Rel Sq Rel RMSE RMSE log δ &lt; 1.25 δ &lt; 1.25 2 δ &lt; 1.25 3</td></tr><tr><td>Godard</td><td>K</td><td>0.295</td><td colspan="2">6.824 11.543</td><td>0.427</td><td>0.638</td><td>0.800</td><td>0.884</td></tr><tr><td>Godard pp</td><td>K</td><td>0.222</td><td>3.121</td><td>9.993</td><td>0.374</td><td>0.659</td><td>0.811</td><td>0.900</td></tr><tr><td>Godard resnet pp</td><td>CS + K</td><td>0.191</td><td>2.923</td><td>9.269</td><td>0.313</td><td>0.700</td><td>0.871</td><td>0.942</td></tr><tr><td>Ours non-shared</td><td>K</td><td>0.229</td><td colspan="2">3.501 10.286</td><td>0.369</td><td>0.655</td><td>0.825</td><td>0.908</td></tr><tr><td>Ours non-shared pp</td><td>K</td><td>0.218</td><td colspan="2">3.099 10.086</td><td>0.361</td><td>0.661</td><td>0.827</td><td>0.905</td></tr><tr><td>Ours non-shared resnet pp</td><td>CS + K</td><td>0.176</td><td>2.216</td><td>8.721</td><td>0.283</td><td>0.738</td><td>0.893</td><td>0.952</td></tr><tr><td>Godard crop</td><td>K</td><td colspan="3">0.2165 3.1826 10.126</td><td>0.369</td><td>0.665</td><td>0.822</td><td>0.903</td></tr><tr><td>Godard pp crop</td><td>K</td><td colspan="3">0.2093 2.9135 9.820</td><td>0.354</td><td>0.673</td><td>0.930</td><td>0.910</td></tr><tr><td>Godard 70 epoch crop</td><td>K</td><td colspan="3">0.2149 3.2421 9.765</td><td>0.363</td><td>0.675</td><td>0.939</td><td>0.910</td></tr><tr><td>Godard resnet pp crop</td><td>CS + K</td><td colspan="3">0.1812 2.4501 9.132</td><td>0.301</td><td>0.707</td><td>0.871</td><td>0.942</td><td>Lower is better</td></tr><tr><td>Ours non-shared crop</td><td>K</td><td>0.2123</td><td colspan="2">3.129 10.050</td><td>0.353</td><td>0.678</td><td>0.836</td><td>0.916</td></tr><tr><td>Ours non-shared pp crop</td><td>K</td><td colspan="3">0.2043 2.8846 9.839</td><td>0.341</td><td>0.678</td><td>0.841</td><td>0.921</td><td>Higher is better</td></tr><tr><td>Ours non-shared 70 epoch crop</td><td>K</td><td>0.2074</td><td>2.907</td><td>9.933</td><td>0.351</td><td>0.673</td><td>0.836</td><td>0.915</td></tr><tr><td>Ours non-shared resnet pp crop</td><td>CS + K</td><td>0.1649</td><td>1.982</td><td>8.570</td><td>0.271</td><td>0.749</td><td>0.900</td><td>0.957</td></tr><tr><td>Godard crop&amp;cap 50m</td><td>K</td><td>0.198</td><td>2.277</td><td>7.119</td><td>0.336</td><td>0.707</td><td>0.820</td><td>0.919</td></tr><tr><td>Godard pp crop&amp;cap 50m</td><td>K</td><td>0.193</td><td>2.132</td><td>6.294</td><td>0.321</td><td>0.714</td><td>0.858</td><td>0.927</td></tr><tr><td>Godard 70 epoch crop&amp;cap 50m</td><td>K</td><td>0.199</td><td>2.348</td><td>6.992</td><td>0.334</td><td>0.713</td><td>0.861</td><td>0.924</td></tr><tr><td>Godard resnet pp crop&amp;cap 50m</td><td>CS + K</td><td>0.162</td><td>1.584</td><td>6.006</td><td>0.262</td><td>0.753</td><td>0.908</td><td>0.961</td></tr><tr><td>Ours non-shared crop&amp;cap 50m</td><td>K</td><td>0.193</td><td>2.165</td><td>6.945</td><td>0.315</td><td>0.715</td><td>0.867</td><td>0.934</td></tr><tr><td>Ours non-shared pp crop&amp;cap 50m</td><td>K</td><td>0.186</td><td>1.981</td><td>6.768</td><td>0.304</td><td>0.721</td><td>0.871</td><td>0.939</td></tr><tr><td>Ours non-shared 70 epoch crop&amp;cap 50m</td><td>K</td><td>0.188</td><td>1.991</td><td>6.792</td><td>0.314</td><td>0.718</td><td>0.869</td><td>0.932</td></tr><tr><td colspan="2">Ours non-shared resnet pp crop&amp;cap 50m CS + K</td><td>0.147</td><td>1.250</td><td>5.526</td><td>0.233</td><td>0.795</td><td>0.929</td><td>0.959</td></tr></table>

Evaluation result on virtual-KITTI. Once our ground truth become dense, our model out performance other models for its sharp and clear boundaries on predicted depth map. Even we crop the black edges of Godard et al (the left 10% of the whole disparity map) in the second row, our model is still superior toGodard et al. (2017) (the state of art unsupervised monocular depth estimation using left and right information only). We evaluate on 3 different methods: naive, crop the black edge, both crop the edge and set the maximun evaluation depth to 50 meters

Table content: <table/>

Peer review (iclr2019):
 This paper proposes a method for learning to perform monocular depth estimation, using calibrated pairs of stereo images - without supervision from ground truth depth maps. It is heavily based on Godard et al., CVPR 2017, and focuses on sharp (*clear*) reconstruction of occluding boundaries. My first problem with this paper is the poor quality of English, which makes it difficult to understand. I do not think this submission can be accepted for publication at ICLR, if only because of this. Technically speaking, if I understand correctly, the key idea is: At each optimization step, consider the predicted depth map. The points that are predicted to be visible in only one of the two images according to this depth map are *disabled* during gradient back propagation, so that they do not influence the network optimization. The motivation is that the loss function assumes that each pixel in the first image is also visible in the corresponding second image of the stereo pairs - which is not true along the occluding boundaries. There is another technique, which (I think) consists in flipping the images and their roles, but I do not understand why it should improve learning - besides providing more training data. The proposed method sounds weird to me: If I understand correctly, there is no guarantee that learning will converge to anything meaningful. For example, the optimization could introduce more occlusions than there really is, to disable the influence of more pixels without being penalize?