When the Sun Goes Down: Repairing Photometric Losses for All-Day Depth Estimation
Authors: Madhu Vankadari
Stuart Golodetz
Sourav Garg
Sangyun Shin
Andrew Markham
Niki Trigoni
Venue: CoRL 2022 Poster (2022)
Keywords: All day depth estimation
3D reconstruction
Photometric losses

Abstract: Self-supervised deep learning methods for joint depth and ego-motion estimation can yield accurate trajectories without needing ground-truth training data. However, as they typically use photometric losses, their performance can degrade significantly when the assumptions these losses make (e.g. temporal illumination consistency, a static scene, and the absence of noise and occlusions) are violated. This limits their use for e.g. nighttime sequences, which tend to contain many point light sources (including on dynamic objects) and low signal-to-noise ratio (SNR) in darker image regions. In this paper, we show how to use a combination of three techniques to allow the existing photometric losses to work for both day and nighttime images. First, we introduce a per-pixel neural intensity transformation to compensate for the light changes that occur between successive frames. Second, we predict a per-pixel residual flow map that we use to correct the reprojection correspondences induced by the estimated ego-motion and depth from the networks. And third, we denoise the training images to improve the robustness and accuracy of our approach. These changes allow us to train a single model for both day and nighttime images without needing separate encoders or extra feature networks like existing methods. We perform extensive experiments and ablation studies on the challenging Oxford RobotCar dataset to demonstrate the efficacy of our approach for both day and nighttime sequences.

When the Sun Goes Down: Repairing Photometric Losses for All-Day Depth Estimation
Madhu Vankadari
Stuart Golodetz
Sourav Garg QUT, Australia
Sangy un Shin University of Oxford
University of Oxford
University of Oxford
Andrew Markham
Niki Trigoni University of Oxford
University of Oxford
Abstract: Self-supervised deep learning methods for joint depth and ego-motion estimation can yield accurate trajectories without needing ground-truth training data. However, as they typically use photometric losses, their performance can degrade significantly when the assumptions these losses make (e.g. temporal illu- mination consistency, static scene, and the absence of noise and occlusions) are violated. This limits their use for e.g. nighttime sequences, which tend to contain many point light sources (including on dynamic objects) and low oggall-o-noii ratio (SNR) in darker image regions. In this paper, we show how to bination of three techniques to allow the existing photometric losses to work for both day and nighttime images. First, we introduce per-pixel neural intensity transformation to compensate for the light changes that occur between successive frames. Second, we predict a per-pixel residual flow map that we use to correct the reprojection correspondences induced by the estimated ego-motion and depth from the networks. And third, we denoise the training images improve the ro- bustness and accuracy of our approach. These changes allow us train single model for both day and nighttime images without needing separate encoders extra feature networks like existing methods. We perform extensive experiments and ablation studies on the challenging Oxford RobotCar dataset to demonstrate the efficacy of our approach for both day and nighttime sequences.
1 Introduction
An ability to capture 3D scene structure is crucial for many applications, including autonomous driving [1], robotic manipulation [2], and augmented reality [3]. Many methods use LiDAR or fixed- baseline stereo to acquire the depth needed to reconstruct a scene, but researchers have also long been interested in estimating depth from monocular images, driven by the ubiquity, low cost, low power consumption and ease of deployment of monocular cameras. By contrast, LiDAR can be power- hungry, and stereo rigs must be calibrated and tine-synchronised to achieve good performance.
Multi-view monocular depth estimation approaches have long used variable-baseline stereo over multiple images to recover depth [4, 5]. Mcanwhile, progress in deep learning has opened up the additional possibility of estimating depth from a single monocular image. Deep learning methods for depth estimation can be broadly divided into two types, namely supervised methods [6, 7]. and self/unsupervised methods [8, 9, 10). Typically, supervised approaches have achieved very good results for the dstaset(s) on which they are trained, but their need for ground-trute information during training has often hindered their deployment in new domains.
By contrast, self/unsupervised methods have typically adopted the use of a geometry-based loss function, inspired by the strong physical principles of traditional methods [11, 12]. This loss function commonly referred to as the photometric or appearance loss, and is based on the assumptions that (i) the scene is static (i.e. contains no moving objects), (ii) the illumination in the scene is diffusive
Corresponding author: madhu. Maanaddariess ox. uk
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

(a]
(b)
Figure 1: (a) The challenges posed by nighttime images: (1) low visibility and noise (patch en- hanced for better readability); (2) moving light sources with saturating image regions; (3) point light sources; (4) extreme motion blur. (b) Despite these adverse conditions, which violate the assump- tions made by the photometric loss, our method can successfully estimate accurate depth maps.
(i.e. there are no specular reflections) and temporally consistent (i.e. the pixels to which any scene point projects in any two consecutive frames have the same intensity), and (iii) the images are free of noise and occlusions [11, 12, 13, 14]. In practice, many of these assumptions are at least partly false, which can lead to errors in the estimated depth: scenes are quite likely to contain dynamic objects (e.g. cars, cyclists and pedestrians, in an outdoor driving scenario), surface materials are rarely fully diffusive, and occlusions are common. During the day, is somewhat reasonable assume that the illumination is moderately temporally consistent for image sequences captured outdoors, as the sun is by far the dominant light source in that case, and the light it casts changes only slowly over time; however, at night, the numerous point light sources that are typically turned on after dark (e.g. car headlights, lamp posts, etc.) can cause the illumination to change drastically from one frame to the next. night, also, the motion blur associated with the movement of dynamic objects in the scene (including the ego-vehicle) becomes worse, owing to the longer exposure times typically used when capturing nighttime images [15, 16], and the signal-to-noise ratio of the (darker) images becomes much lower than it would be during daytime. Such issues, illustrated in Figure inhibit the straightforward use of deep networks based on photometric loss for nighttime sequences.
In this paper, we address this problem by directly targeting violations of the temporal illumination consistency, static scene and noise-free free assumptions on which the photometric loss relies. As shown by our day and night results in Table I, these three together account for much of the discrepancy in performance between daytime and nighttime. A lack of temporal illumination consistency caused by point light sources in the scene can cause pixels to be incorrectly matched between consecutive frames. To rectify this, we propose novel per-pixel neural intensity transformation that learns to compensate for these light sources (see $3.2). Whilst conceptually straightforward, this approach surprisingly effective, as our results in $4 demonstrate. Interestingly, they also show that it is able to operate well over wide (motion parallax) baselines, allowing us to leverage the better depth estimation performance that wider baselines offer. To correct for dynamic objects in the scene, as well motion blur, we predict a per-pixel residual flow map (see $3.3) that we use correct the reprojection correspondences induced by the estimated ego- ego-motion and depth from the networks. This improves depth estimation performance at any time of day (see $4), but has additional theoret- ical benefits for nighttime sequences because of the greater motion blur from which they typically suffer. Lastly, we robustify our approach against noise by incorporating Neighbour2Neighbour [17], denoising module, in our photometric loss formulation (see $3.4).
a
2 Related Work
Estimating depth from images has a long history in computer vision. Several methods use either
stereo images [18, 19, 20], or two or more images taken from different viewing angles [21, 22, 23]. We try to solve this problem using single monocular image, without any constraints on the scene of interest. Various methods have addressed this problem using supervised learning [6, 7, 24, 25, 26]. However, it is infeasible to have ground-truth depth maps for training on every scene, which limits the application of these methods and helps motivate unsupervised solutions to this problem.

---

Unsupervised Methods: Garg et al. [8] proposed a geometry- baseed loss function to train network in a completely unsupervised fashion using a pair of stereo images. Monodepth [27] improved this by using differentiable image warping [28] and structural similarity-based [29] image comparison loss. SfMLearner [30] used only monocular images to jointly learn depth and ego-motion.
was further improved by combining stereo and monocular losses [31, 32]. GeoNet [33] and EPC [34] learnt per-pixel optical flow maps along with depth and ego-motion to mitigate the effect of moving objects. Some GAN-based methods also exist [10, 35, 36). Monodepth2 [37] extended Monodepth to the temporal domain, and proposed architectural changes and robust loss functions. More broadly, recent years have also seen a wide range of other advances in depth estimation, c.g. changes to the network architecture [38, 39, 40, 41], changes to the training strategy [42], the addition of extra loss functions [43], and better handling of dynamic objects [44].
coiminnationIInariinn Feature Learning: Illumination inconsistency is broader problem, e.g- in visual relocalisation [45] and 3D reconstruction [46]. In [47], canonical reference image is estimated using image translation and used in a downstream pose estimation task. In [48], a sparse set of keypoints with illumination-invaiian descriptors are learned from images captured at different times of day or in different weather conditions. In [49], learnt illumination-invarian features are used to compute an error metric for optimising the camera pose. Recently, many other methods [50, 51, 52, 53, 54] have targeted sparse keypoint-based relocalisation using learnt illumination- invariant feature descriptors. By contrast, in this work, we aim to solve the per-pixel depth estimation problem. Dense feature learning is also used for daytime depth estimation by [43].
All the methods above are trained using photometric loss as the main supervi- sion signal, and with an assumption of temporal illumination consistency, which is not valid at night. A few methods, such as DeFeat-Nee [55]. ADFA [56] and Nighttime stereo [14], have explored how to estimate depth from nightrime RGB images. DeFeat-Net [55] learns n-dimensional deep feature representations (assumed to be illumination-inaal invariant) using pixel-wise contrastive loss. The fea- ture maps are simultaneously used along with the images for photometric loss calculation during training. ADFA [36] mimics a daytime depth estimation model by learning new encoder that can generate 'day-like' features from nighttime images using a domain adaptation approach. Instead of feature translation as in [56], the authors in [14] propose a joint network for image translation and stereo image-based depth estimation. Recently, photometric loss has again been used RNW [57] with an image enhancement module and a GAN-based depth regulariser. Liu et al. [58] divided the day and nighttime images into fiew-invariant and variant feature maps using separate encoders, and used the view-invariant information for depth estimation. All these methods either need two sepa- rate encoders for day and nighttime images [56, 58, 57], or need to learn an Illumination-inaaiiae feature space [55]. By contrast, our method learns to estimate depth in a completely self-supervised fashion, without needing stereo images, ground- ground-truth depth or any additional feature learning.
Nighttime Methods:
3 Method
3.1
Baseline Method
Existing photometric loss methods typically use two networks, a depth network (or DepthNet) and a motion network (or MotionNer). The DepthNer takes an individual colour image as input, and is used to predict depth image D2 for each colour image T, in the input sequence. The MotionNer takes a consecutive pair of images and Ic+1 as input, and is used to output the ego-motion Tr, of the camera between them. The estimated depth and ego- motion can be used to reproject pixel Dr -'kk---,,--
[u, u]" in frame 4; into 4+1 via (u) form of u, K E R3x3 encodes the camera intrinsics, and (u) E R3 is the homogeneous form of Vu(u) E RR2 2D point in the image plane of 4++1 (which may or may not lie within the bounds of the actual image). This can be used to reconstruct an image T/ by sampling from I++1 around the reprojected points, using bilinear interpolation [28] to achieve a smoother result. Formally,
û, in which ú is the homogeneous
T((1)= interpolate (141, Vi(u)) if M, otherwise,
(1)
3

---

Deyst.Nes
Depth Map Dr
Denctiing
Danoliing
View
T
411
MoliomNel
Fraunity Troncomm
I
C2
Lightchangge
Figure 2: The architecture of our proposed method (see $3 for details).
in which M, {u ((V((u)) E (((++1)) is the set of pixels whose reprojections into I++1, when rounded the nearest pixel using p. falls within the image domain Q2 (1)++)) The reconstructed image 7 then be compared to the original image
calculate the loss values needed for training. The loss we target, namely photometric loss, has been used by many recent deep learning- based depth estimation techniques such as, [8, 31, 30, 36]. It is normally calculated as convex combination of pixel-wise difference and single-scale structural dissimilarity (SSIM) [29]. via
[M EUEM. (a (01- +(1 -1 a:) [L((u) I((w)))
2
Most existing unsupervised methods (e.g. [30, 31, 37, 38)) use this as the backbone of their formu- lation. To ensure fair comparison with current nighttime state-of-the-ar methods [55, 58, 57], we base our modifications this paper on Monodepth2 [37], a commonly used baseline.
3.2 Lighting Change Compensation
The numerous point light sources that are typically turned after dark (e.g. car headlights, lamp posts, etc.) can cause the illumination of a scene to change significantly from frame 7, to frame 1++1- Moreover, when light source moves with the camera (e.g. car headlights), this can lead to large holes in the estimated depth in front of the ego-vehicle [55, 58]. To compensate for the illumination changes, we estimate a per-pixel transformation that, when applied to I++1, can mitigate the changes in lighting that have occurred since T. We draw some inspiration from [59, 60, 61], which use single whole-image transformation based two scalar values to compensate for the difference in exposure time between pair of images, as this creates roughly uniform intensity changes over the entire image. However, in our case, the intensity changes are far from uniform over the image, owing to both the motions of the ego-vehicle and other objects in the scene, and the distances between the ego-vehicle and static point light sources. For this reason, we propose a per-pixel formulation here.
Our approach starts by passing the features produced by the last convolutional layer of the Motion- Net through lighting change d to estimate per-pixel change images, C₁ and B, (see Figure 2). These (respectively) aim to capture the per-pixel changes in contrast (scale) and bright- ness (shift) that have occurred between the two input frames. As shown in Figure 4, the brightness image B, broadly captures the extra light added to the image by e.g. vehicle headlights, and the con- trast image C, broadly captures the changes in ambient light due to the motion of the ego-vehicle towards or away from point light sources such as street lamps. We use these images to transform the reconstructed image via C, 1 + B, which denotes the Hadamard product.
Similar to the DepthNet decoder, but without skip connections. See supplementary material for details.

---

3.3 Motion Compensation
The standard photometric loss makes use of correspondences {u + Vi(u): u E M3 between consecutive frames that have been established via reprojection, based on the estimated ego-motion and depth. Assuming that (i) the ego-motion and depth have been estimated well, (ii) the scene is static, and (iii) there is minimal motion blur, these correspondences will broadly match those that would have been established had we used the ground-truth optic flow (x(-) from frame & to frame t+1. However, if objects move with respect to the background scene, or anything visible in the image moves with respect to the ego-camera (which can cause motion blur), then these correspondences may be incorrect. To correct them, we predict residual flow map RG such that for each pixel u, (u) E IR? is an estimate of (u + +. (u)) - Vi(u). We can then add R₂ (u) to Vu(u) to obtain a potentially more accurate correspondence for use in reconstructing I via Equation 1.
Some methods [34, 33, 44] already exist that predict residual flow for daytime images by using separate encoder-decoder network computationally intensive image warping- based bilinear inter- polation for supervision. By contrast, we estimate residual flow using an efficient sparsity-based formulation. This involves introducing a residual flow decoder that takes the features of the final convolutional layer of the MotionNet as input and the features previous layers in the MotionNer via skip connections, and outputs residual flow maps {Rr.. : s E {0, 1, 2, 3 at four different scales (each R has width and height that is 1/2" that of Lt, and Re.0 Rd).
There is no direct supervision available to learn the residual flow maps. For this reason, we choose instead to encourage sparsity in the residual flow estimates, so that the estimated depth and ego- motion can explain the majority of the scene, and the left-over can be explained by the residual flow maps. To achieve this, we adopt the sparsity loss from [44], i.c.
= 3 ( EuEn(u.)) VI (u))((R)))
(3)
in which It., is downsampled version of I, scale 8, and [)) is the spatial average of the abso- lute residual flow map | By contrast with [44], here we introduce a normalising factor of 1/2* at each scale, since the original loss was for scene flow, where the flow magnitude is independent of the resolution of the flow maps, which is not the case for the 2D residual flow we consider.
3.4 Image Denoising
Image noise is yet another key factor that affects the performance of the photometric loss, especially in darker regions of the image that typically have a low SNR. Handling this noise is crucial, as photo- metric loss is the only training signal. To denoise the images, we used Neighhouur2eighhbou [17], a -late-of-the-art unsupervised model trained on ImageNet with zero-mean Gaussian noise. The stan- dard deviation was varied from 5 to 50 during training. We denoise the images before feeding them to the network. This either be done at both training and test times, solely training time (for calculating the loss). We chose the latter, as the former has two potential disadvantages: (i) it can significantly add to the computational burden at runtime, slowing down the depth estimation, and (ii) any errors in the denoising process can lead to downstream errors in the depth maps, even though the depth estimation model itself might have been trained well. By contrast, restricting denoising to training time has the advantage of allowing us to make the depth and motion networks robust to noise by training them on the original noisy images, but supervising with the denoised images.
3.5 Full Pipeline
We can now formulate our full pipeline, which takes two consecutive images I, and I
D. -(()), fn ME (.,, d.t1)), Th.41 =DD(NN)
R, RFF 115 NH), (.), LCD(fN)
I, (L,,)), =+1=DN(L I reconstruct(Iuti. V. Ro), 1, C. O 1/+ +B.
(4)
(1) 1. Σncm, (o (w. + (1 a) [(u) (u) Ze(1))
McT
5

---

where D is the DepthNet, 1: denotes the first n layers of the N-layer MotionNer encoder, MD is the MotionNet decoder, RFFD is the residual flow decoder, LCD is the lighting change decoder, and DN is the denoiser [17]. The reconstruct function reconstructs T₁ as per Equation 1, and Vi denotes the reprojected pixels from I, to 11-- At frame I, the DepthNel is applied to I, to estimate a per-pixel depth map D., and the MotionNer is applied to both 1, and 4++1 estimate the ego-motion Te.+1 from frame to frame t 1. During training, the features from the last convolutional layer of the MotionNet are passed through the lighting change decoder to estimate the per-pixel lighting changes that have occurred between the frames (see $3.2). A separate residual flow decoder takes both these last layer features and the features of earlier layers in the MotionNet (via skip connec- tions) to estimate a per-pixel residual flow map Rg (see $3.3) that we use to correct the reprojection correspondences induced by the estimated ego-motion and depth from the networks. To calculate the photometric loss, we (i) denoise 4, and 4+1 with a state-of-the-arr denoising module [17] to produce I, and I+ (ii) reconstruct I from Iz+1 using reprojection correspondences that have been corrected using the residual flow; (iii) correct Ij for the estimated lighting changes between the frames to produce I;; and then (iv) calculate the usual photometric loss between I, and I.
Making the Pipeline Bidirectional. Monodepth2 [37] calculates its photometric loss not only forwards, from Ie to I,+1, but also backwards, from I, to It-1, before combining the losses. This allows us use the idea of minimum reprojection error to account for occluded pixels, and we do the same. We also adopt the auto-masking losses from Monodepth2 [37], which we call LG" as even though our method can cope with moving objects, it is very difficult to use parallax to disentangle the motion of objects that are moving in the same direction and at the same speed as the ego-vehicle.
We further include the commonly used edge-aware gradient smoothing loss Lg [27] to maintain spatial smoothness over the estimated depth maps. Our final loss (1) then becomes
( min (1.2. 0. 0 a- 1..4 + Ar [1,0 + 1402 + > L0
(5)
in which +/- denote the forward/backward versions of the losses, and Ar, X2 E R are the weights.
4 Experiments
In $4.1, we compare our depth estimation performance to number of state-of-the-art approaches in variety of different daytime and/or nighttime contexts. In $4.2, we present a study on the effect of parallax to help explain the importance of our neural intensity transformation module. Finally, in $4.3, we perform an ablation study to analyse the contributions made by the three individual components of our approach. Further experiments can be found in the supplementary material.
4.1 Depth Evaluation
We compare with 4 state-of-the-art unsupervised monocular methods: Monodepth2 [37], DeFeat- [58] and RNW [57] (see Figure 3 and Table 1). The quantitative
Net [55], ADDS-Depth-Nipe evaluation uses the eттor and accuracy metrics from [6], as detailed in the supplementary material. We tested our model with 3 different data variations: day only (d), night only (n), and a mix of day and night (d&n). Monodepth2 [37] can be trained with all 3 configurations, but has already been outperformed by DeFeat-Nett [55] in the d&n setting. For the d and n settings, we outperform it by significant margin in both error and accuracy (see Table 1). DeFeat-Net [55] and ADDS- Degth-Night [58] were originally trained with d&an configuration. We evaluated the pre- models they released on our test split. Our method outperforms both methods by a significant margin on the nighttime sequences (see Table 1). Please note that we do not use any additional feature representation- based losses as used in DeFeat-Net [55], or paired day and night images as used in ADDSSDDepth/l data only. As per Figure 3, our depth estimation results are sharper and better able to preserve edges than the competing methods. We also found that using a longer baseline improves depth estimation performance. However, naïvely using wider baseline without also using our neural intensity transform can lead to severe decrease in accuracy, particularly for nighttime images.
[58]. RNW [57] is also built on Monodepth2, but targets nighttime

---

Test
Method
Train
Abs. Rel.
Sq. Rel.
RMSE
Log RMSE
6 6 1.25
6 < 1.25
8 1,25
Monodepth2 [37]
d 0.219
4.525
7.641
0.285
0.679
0,862
0,930
Ours
d
0.191
1.710
6.158
0.253
0.713
0,904
0,962
Day
DeFeat-Net [55]
& 0.247
2.980
7.884
0.305
0,650
0.866
0,943
RNW 157]
d &
0.297
2.608
7.996
0.359
0,431
0,773
0,930
ADDS-Depth-Niht [58]
d &
0.239
2.089
6.743
0.295
0.614
0.870
0,950
Ours
d &
0.176
1.603
6.036
0,245
0.750
0.912
0,963
Monodepth2 [37]
n
0.453
21.310
11.420
0.444
0,700
0.873
0,930
RNW MCIE SBM
0.350
7.934
8.994
0.407
0.674
0.861
0,922
Nighht
Ours
0.186
1.656
6,288
0.248
0.728
0.919
0.969
DeFeat-Net [55]
d & 0.334
4.589
8.606
0.358
0,586
0,827
0.911
ADDS-Depth-Nighe [58]
d
0.287
2.569
7.985
0.339
0,490
0,946
RNW [57]
d &
0.185
1.710
6.549
0,262
0.733
0.910
0.960
Ours
d & n
0.174
1.637
6,302
0.245
0.754
0.915
0.964
Table 1: A quantitative comparison of our method. The results of Monodepth2 [37] are reported after retraining it. Those of DeFeat-Net [55] and ADDS- BDDS-Depth-Nighu [58] are reported using the checkpoints from their public repositories. The evaluation uses a maximum depth of 50m. Under- lined methods use daytime images as main supervision for regularisation losses.
Input RGB image
LIDAR Ground Trush
Monodesth22
DeFeat-Net
ADDS SepSDDethNett
Ours
Figure 3: A qualitative comparison of our proposed method with the state of the art.
4.2 Effect of Parallax
To better understand how depth estimation performance is affected by increasing the average paral- lax (metric separation) between the images we use to calculate the photometric loss, we constructed a new nighttime training split by increasing the intra-triplet stride (see supplementary material) to 2, which increased the average parallax between the images from 0.353m to 0.706m. Without our neural intensity transformation, the depth estimation performance significantly decreased compared the original nighttime training split (see the difference between the RMSES of the baseline in the top and bottom parts of Table 2). A key cause of this in night images is likely the headlights of the ego-vehicle, which can cause the pixel intensities to change drastically between frames. However, with our neural intensity transformation, the depth estimation performance was found to instead in- crease, which we hypothesise to be because by compensating for the lighting changes, we make possible to exploit the stronger supervision that can be offered by a wider baseline.
4.3 Ablation Study
Lighting Change Compensation. In Figure 4(a), we show several reference images and their light- ing change maps. The intensity changes are non-uniform, so we cannot use the existing correction approaches from [59, 60, 61]. We also observe that our method is able to elearly disentangle both the changes in ambient light resulting from movement towards/away from point light sources (captured by C₂) and the additional light added the road pixels in the images by the eeo-vehicle headlights (captured by B,). Our neural intensity transform significantly reduces the RMSE compared to the baseline (see Table 2), and is also able to fill in holes in front of the eeo-ve (see Figure 3).
7

---

Stride
Method
Abx. Rel.
Ret.
RMSE
Log RMSE
1.25
6 1.25²
6 1.25"
Baseline
0.266
5.647
6.305
0.331
0.759
0,9013
0,947
w/ NIT
0.190
1.824
4,848
0.257
0,763
0.919
0.965
Denoising Full Model
1.256
0.224
0.154
1,174
4.120
0.216
0,811
0.939
0.976
Baseline
0.602
63,914
14.726
0.467
0.785
0.902
0.939
2
w/ NIT 0.169
1.727
4.693
0,236
0.812
0.929
0.967
Full Model 0.131
0,926
3.731
0.188
0.949
0,980
Table 2:
Ablation study showing the importance of different modules in system where NIT
stands for neural intensity transformation. The maximum evaluation depth was set to 30m for this study. 'Stride' denotes intra-triplet stride (see supplementary material).
B.
0.01 WET only
g0409
40000
640
300 125
350
200
(c)
Figure 4: Visualisations of (a) estimated light changes; (b) residual flow and estimated depth; (c) the effects of denoising on the training loss over time. Please refer to $4.3 for more details.
Motion Compensation. In Figure 4(b), we show several reference images and their residual flow and depth maps. In the second column, one can clearly see that our method able to distinguish pixels on moving objects such as cars and pedestrians from static pixels. This effect can be observed for both daytime and night-time images, showcasing the generality of approach through single unified training pipeline. In Table 2, it can be seen that correcting the reprojection correspondences using the residual flow map we predict leads to significant improvement in accuracy.
Image Denoising. Denoising the images while calculating the training loss should idcally reduce the ambiguity in establishing pixel correspondences between the images, giving robust supervision signal for training our system and thereby achieving lower errors and higher accuracy. This effect can be clearly seen in the training error plot shown in Figure 4(c), where we compare base- line+NIT model with and without denoising. The denoising results in much more accurate depth maps, improving both the RMSE and accuracy metrics as shown in Table 2.
Limitations
Like most stereo approaches (variable-baseline like ours, or fixed-baseline with rigid stereo rig), our method struggles to preserve the detail of distant parts of the scene because of limited parallax. also struggles to recover structural detail from very dark image regions (e.g. see Figure 1 (b)). Furthermore, we currently approximate the relationship between pixel intensities and light intensity as linear for simplicity. This can be better approximated by using the inverse camera-response function. Finally, whilst our method copes with most motion patterns, can struggle to estimate the motion of objects moving with the same speed the ego-vehicle and in the same direction.
Conclusions
In this paper, we propose a self-supervised method to learn a single model estimate depth maps from monocular day and nighttime RGB images. By compensating for the illumination changes that can occur from one frame to the next, we enable accurate nighttime depth estimation in non- uniform lighting conditions. Moreover, by predicting per-pixel residual flow and using it to correct the reprojection correspondences induced by the estimated ego-motion and depth, we improve our method's ability to cope with both moving objects in the scene and motion blur. Finally, by denoising the input images prior to calculating the photometric loss, we improve the loss's ability to provide strong supervision signal, making the entire system more robust and accurate.

---

Acknowledgments
This work was supported by AWS via the Oxford-Singapore Human-Machine Collaboration Pro- gramme, and EPSRC via ACE-OPS (EP/S030832/I). The authors would also like thank the anonymous reviewers for their helpful comments.
References
[1] E. Yurtsever, Lambert, Carballo, and K. Takeda. A Survey of Autonomous Driving: Common Practices and Emerging Technologies. IEEE Access, 8:58443-58469, 2020.
[2] J. Sanchez, J.-A Corrales, B.-C. Bouzgarrou, and Y. Mezouar. Robotic manipulation and sensing of deformable objects domestic and industrial applications: a survey. I/RR, 37(7): 688-716, 2018.
[3] M. A. Livingston, Z. J. Swan, and H. Smallman. Indoor vs. Outdoor Depth Perception for Mobile Augmented Reality. In IEEE VR, pages 55-62, 2009.
[4] R. A. Newcombe, S. J. Lovegrove, and A. J. Davison. DTAM: Dense Tracking and Mapping In ICCV, pages 2320-2327, 2011.
in
[5] K. Wang and S. Shen. MVDepthNet: Real-time Multiview Depth Estimation Neural Network. In 3DV, pages 248-257, 2018.
[6] D. Eigen, C. Puhrsch, and R. Fergus. Depth Map Prediction from a Single Image using a Multi-Scale Deep Network. NeurIPS, pages 2366-2374, 2014.
[7] F. Liu, C. Shen, and G. Lin. Deep Convolutional Neural Fields for Depth Estimation from Single Image. In CVPR, pages 5162-5170, 2015.
[8] R. Garg, K. BG, G. Cameiro, and Reid. Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue. ECCV, pages 740-756, 2016.
[9] H. Zhan, R. Garg, C. S. Weerasekera, K. Li, H. Agarwal, and I. Reid. Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction. In CVPR, pages 340-349 -004,,99400 2018.
[10] Y. Almalioglu, M. R. U. Saputra, P. P. B. de Gusmão, A. Markham, and N. Trigoni. GANVO: Unsupervised Deep Monocular Visual Odometry and Depth Estimation with Generative Ad- versarial Networks, ICRA, pages 5474-5480, 2019.
[11] C. Kerl, J. Sturm, and D. Cremers. Dense Visual SLAM for RGB-D Cameras. In IROS, pages 2100-2106, 2013.
[12] C. Kerl, J. Sturm, and Cremers. Robust Odometry Estimation for RGB-D Cameras. In ICRA, pages 3748-3754, 2013.
[13] A. I. Comport, E. Malis, and P. Rives. Accurate Quadrifocal Tracking for Robust 3D Visual Odometry. In ICRA, pages 40-45, 2007.
[14] A. Sharma, L.-F. Cheong. L. Heng, and T. Tan. Nighttime Stereo Depth Estimation using Joint Translation-Stereo Learning: Light Effects and Uninformative Regions. In 3DV, pages 23-31, 2020.
[15] T. Portz, L. Zhang, and H. Jiang. Optical Flow in the Presence of Spatially Varying Motion Blur. CVPR, pages 1752-1759, 2012.
[16] A. Rav-Acha and S. Peleg. Restoration of Multiple Images with Motion Blur in Different Directions. In WACV, pages 22-28, 2000.

---

[17] T. Huang, S. Li, X. Jia, H. Lu, and J. Liu. Neighbor2Neighbor: Self-Supervised Denoising from Single Noisy Images. In CVPR, pages 14781-14790, 2021.
[18] D. Scharstein and Szeliski. Taxonomy and Evaluation of Dense Two-Frame Stereo Cor- respondence Algorithms. IJCV, 47(1-3)77-42, 2002.
[19] D. Scharstein and C. Pal. Learning Conditional Random Fields for Stereo. In CVPR, pages 1-8, 2007.
[20] L. Zou and Y. Li. A Method of Stereo Vision Matching Based on OpenCV. In ICALIP, pages 185-190, 2010.
[21] J. L. Schönberger and J.-M. Frahm. Structure-from-Motiie Revisited. In CVPR, pages 4104- 4113, 2016.
[22] Y. Dai. H. Li, and M. Hc. Projective Multiview Structure and Motion from Sement-WWsst Factorization. TPAMI, 35(9):2238-2255 2013.
[23] Yu and D. Gallup. 3D Reconstruction from Accidental Motion. CVPR, pages 3986-3993, 2014.
[24] L. Ladicky, J. Shi, and M. Pollefeys. Pulling Things out of Perspective. In CVPR, pages 89-96, 2014.
[25] N. dos Santos Rosa, V. Guizilini, and V. Grassi Jr. Course-to-Continuuut Enhancing Monoc- ular Depth Estimation using Occupancy Maps. In ICAR, pages 793-800, 2019.
[26] H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao. Deep Ordinal Regression Network for Monocular Depth Estimation. In CVPR, pages 2002-2011, 2018.
[27] C. Godard, O. Mac Aodha, and G. J. Brostow. Unsupervised Monocular Depth Estimation with Legt-Right Consistency. In CVPR, pages
2017.
[28] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. Spatial Transformer Net- works. In NeurlPS, pages 2017-2025, 2015.
[29] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image Quality Assessment: From Error Visibility to Structural Similarity. TIP, (3(4))600-612, 2004. 2004
[30] Zhou, M. Brown, N. Snavely, and D. G. Lowe. Unsupervised Learning of Depth and Ego- Motion from Video. In CVPR, pages 1851-1860, 2017.
[31] M. Babu V., K. Das, A. Majumdar, and S. Kumar. UnDEMoN: Unsupervised Deep Network for Depth and Ego-Motion Estimation. In IROS, pages 1082-1088, 2018.
[32] R. Li, S. Wang, Z. Long, and D. Gu. UnDeepVO: Monocular Visual Odometry through Unsu- pervised Deep Learning. ICRA. pages 7286-7291, 2018.
[33] Z. Yin and J. Shi. GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose. In CVPR, pages 1983-1992, 2018.
[34] C. Luo, Z. Yang, P. Wang. Wang, W. Xu, R. Nevatia, and A. Yuille. Every Pixel Counts++: Joint Learning of Geometry and Motion with 3D Holistic Understanding. TPAMI, 42(10): 2624 2641, 2019.
[35] F. Aleotti, F. Tosi, M. Poggi, and S. Mattoccia. Generative Adversarial Networks for unsuper- vised monocular depth prediction. In ECCV-W. W, 2018.
[36] M. Vankadari, S. Kumar, A. Majumder, and K. Das. Unsupervised Learning of Monocular Depth and Ego-Motion using Conditional PatchGANs, In IJCAI, pages 5677-5684, 2019.
10

---

[37] C. Godard, O. Mac Aodha, M. Firman, and G. Brostow. Digging Into Self-Supervised Monoc- ular Depth Estimation. In ICCV, pages 3828-3838, 2019.
[38] V. Guizilini, R. Ambrus, S. Pillai, A. Raventos, and A. Gaidon. 3D Packing for Self Superviuee Monocular Depth Estimation. In CVPR, pages 2485-2494, 2020.
[39] X. Lyu, L. Liu, M. Wang. X. Kong, Liu, Liu, X. Chen, and Y. Yuan. HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation. In AAAI, pages 2294-2301, 2021.
[40] Yan, H. Zhao, P. Bu, and Y. Jin. Channel- Wise Attention- Basedd Network for Supervised Monocular Depth Estimation. In 3DV, pages 464 473, 2021.
[41] T.W.. Hui. RM-Depth: Unsupervised Learning of Recurrent Monocular Depth in Dynamic Scenes. In CVPR, pages 1675-1684, 2022.
[42] A. Petrovai and S. Nedevschi. Exploiting Pseudo Labels in a Self-Supervised Learning Frame- work for Improved Monocular Depth Estimation. In CVPR, pages 1578-1588, 2022.
[43] C. Shu, K. Yu, Z. Duan, and K. Yang. Feature-metric Loss for Self-supervised Learning of Depth and Egomotion. In ECCV, pages 572-588, 2020.
[44] H. Li, A. Gordon, Zhao, V. Casser, and A. Angelova. Unsupervised Monocular Depth Learning in Dynamic Scenes. In CoRL, pages 1908-1917, 2021.
[45] A. Kendall, M. Grimes, and R. Cipolla. PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization, In ICCV, pages 2938-2946, 2015.
[46] M. Bloesch, T. Laidlow, R. Clark, S. Leutenegger, and A. J. Davison. Learning Meshes for Dense Visual SLAM. In ICCV, pages 5855-5864, 2019.
[47] L. Clement and Kelly. How to Train a CAT: Learning Canonical Appearance Transforma- tions for Direct Visual Localization Under Illumination Change. RA-L, 33):2447-24444 2018.
[48] M. Gridseth and T. D. Barfoot. Keeping an Eye on Things: Deep Learned Features for Long- Term Visual Localization. RA-L, 7(2): 2021.
[49] B. Xu, A. J. Davison, and S. Leutenegger. Deep Probabilistic Feature-Metric Tracking. RA-L, 6(1):223-230, 2020.
[50] P. Lindenberger, P.EE. Sarlin, V. Larsson, and M. Pollefeys. Pixel-Perfect Structure-from- Motion with Featuremetric Refinement. In ICCV, pages 5987-5997, 2021.
[51] P.-E. Sarlin, A. Unagar, M. Larsson, H. Germain, C. Toft, V. Larsson, M. Pollefeys, V. Lepetit,
L. Hammarstrand, F. Kahl, and T. Sattler. Back to the Feature: Learning Robust Camera Localization from Pixels to Pose. In CVPR, pages 3247-3257,200
[52] L. von Stumberg* R Wenzel*, Q. Khan, and D. Cremers. GN-Net: The Gauss-Newton Loss for Multi-Weathin Relocalization. RA-L, 5(2):890- 897000089990100 2020.
[53] S. Chen, X. Li, Z. Wang, and V. A. Prisacariu. DFNet: Enhance Absolute Pose Regression with Direct Feature Matching. In ECCV, pages 1-17, 2022.
[54] L. von Stumberg*, P. Wenzel*, N. Yang, and D. Cremers. LM-Reloe: Gevenberg-Marquaae Based Direct Visual Relocalization. In 3DV, pages 968-977, 2020.
[55] J.
Spencer, R. Bowden, and S. Hadfield. DeFeat-Net: General Monocular Depth via Simulta- neous Unsupervised Representation Learning. In CVPR, pages 44402-14413, 2020.
[56] M. Vankadari, Garg, A. Majumder, S. Kumar, and A. Behera. Unsupervised Monocular Depth Estimation for Night-time Images using Adversarial Domain Feature Adaptation. In ECCV, pages 443-459, 2020.

---

[57] K. Wang*, Z. Zhang", Z. Yan, X. Li. B. Xu, J. Li, and J. Yang. Regularizing Nighttime Weirdness: Efficient Self-supervised Monocular Depth Estimation in the Dark. In ICCV, pages 16055-16064, 2021.
[58] L. Liu, X. Song, M. Wang, Y. Liu, and L. Zhang. Self-supervised Monocular Depth Estimation for All Day Images using Domain Separation. In ICCV, pages 12737-12746, 2021.
[59] H. Jin, P. Favaro, and Soatto. Real-Time Feature Tracking and Outlier Rejection with Changes in Illumination. In ICCV, pages 684-689, 2001.
[60] S. Baker and I. Matthews. 221-255, 2004.
20 Years On: A Unifying Framework. IJC,, 56(3):
[61] N. Yang, L. von Stumberg, R. Wang, and D. Cremers. D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry. In CVPR, pages 1281-1292, 2020.
12


Reviews:
[{"review_id": "y-y5mjZjd0c", "reviewer": "Reviewer_BWdU", "summary": "This paper presents three contributions to improve self-supervised depth and ego-motion estimation:\n\n1. a novel per-pixel neural intensity transformation network to compensate for lighting changes; \n2. a residual flow map for each pixel to correct the reprojection correspondences;\n3. Using a Neighbour2Neighbour algorithm to denoise input images", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well written and easy to follow.\n* The lighting change decoder idea seems interesting and useful.\n* The experiments are conducted on the public dataset and tested extensively at different combinations of day and night, with a good separation of training and testing routes.\n* The result shows a consistent improvement compared to baseline methods.\n\n\nWeaknesses:\n* In general, the novelty in this paper is somewhat limited. It was not the first time that motion compensation and image denoising on the inputs are proposed for depth estimation and ego-motion learning.\n* There are a few issues that could be treated as weaknesses. However, they may also be able to be addressed in the revision by the authors. Please take a look at the issues I listed below. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper may have a few limited theoretic novelties, but it shows well-conducted experiments and improvements compared to baseline methods. The paper can be improved if the mentioned issues are addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. Generalisation: The method is trained and tested in the Oxford RobotCar dataset, with a good separation of training and testing routes. However, I am still a bit curious about whether the system trained in the Oxford Robotcar dataset can work well in other environments, day and night.\n2. The assumption of constant illumination is an issue for photometric loss. Some existing works address this issue as well, but instead of using a motion network directly to regress the relative transformation, they formulate it as an optimisation problem. I believe they are also worth being mentioned in the related work section:\n    1. Clement, Lee, and Jonathan Kelly. \"How to train a cat: Learning canonical appearance transformations for direct visual localization under illumination change.\" IEEE Robotics and Automation Letters 3.3 (2018): 2447-2454.\n    2. Bloesch, Michael, et al. \"Learning meshes for dense visual slam.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n    3. Xu, Binbin, Andrew J. Davison, and Stefan Leutenegger. \"Deep probabilistic feature-metric tracking.\" IEEE Robotics and Automation Letters 6.1 (2020): 223-230.\n    4. Gridseth, Mona, and Timothy D. Barfoot. \"Keeping an Eye on Things: Deep Learned Features for Long-Term Visual Localization.\" IEEE Robotics and Automation Letters 7.2 (2021): 1016-1023.\n3. The idea of capturing the per-pixel changes in contrast (scale) and brightness (shift) looks interesting. I appreciate the theortical analysis in the supporting material to prove this assumption. However, it looks to me that there are too many simplifications involved to derive from the Phong illumination model to the proposed linear combination of scale and shift lighting change equation. In particular, the simplification of removing the reflection term looks a bit suspicious. I think a big portion of the lighting changes that occur at night is the reflection on roads and surrounding vehicles.\n4. I am wondering why the proposed method does not utilise geometric consistency. The depth image $D_{t+1}$ from $I_{t+1}$ is not used in the pipeline.\n4. The limitation section is a bit too general. "}}, {"review_id": "jEO4OguLwfn", "reviewer": "Reviewer_YCUH", "summary": "This paper describes an approach to the self-supervised training of deep networks for joint depth and ego-motion estimation that are capable of accurate all-day (i.e. day *and* night) prediction.  The main technical contribution is to identify three assumptions underpinning the use of the photometric loss for self-supervision that are significantly violated during nighttime operation (temporal illumination consistency, static scenes, and absence of noise and occlusions), and then show how to modify standard architectures for monocular depth and ego-motion estimation to correct for these sources of error.  The result is a single network architecture that can be trained and deployed on any combination of day and nighttime image sequences.  The experimental results provide convincing evidence that the proposed system significantly outperforms comparable baselines on both day and nighttime applications, including prior works specifically designed for nighttime use.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In my opinion the paper’s greatest strength is that the technical approach is explicitly grounded in the physics and geometry of image formation.  \n\nFor example, the paper motivates the development of the lighting change compensation module by considering the validity of the “temporal illumination consistency” assumption in daytime operation (where most of the ambient illumination comes from the sun, which is well-approximated as a static source) versus nighttime operation (where much of the ambient illumination comes from traffic lights, streetlamps, and headlights of other vehicles, which are bright point sources that move over time relative to a moving camera).  The proposed illumination compensation method is a simple yet effective linear model based upon a predicted *contrast* map (which is intended to capture *global* changes in illumination caused by motion of the vehicle towards / away from ambient sources such as streetlights) and a predicted *brightness* map (which accounts for the presence of potentially moving *point* sources, such as vehicle headlights).  Similar analyses lead to the motion correction approach (designed to account for motion blur and / or the motion of objects in the scene, such as vehicles) and the denoising approach used to pre-process the training images.\n\nThe design of the proposed approach is thus well-justified, physically grounded, and readily interpretable, all of which are highly desirable properties.  \n\nFurthermore, because the approach is based upon accounting for violations of the assumptions underpinning the photometric loss (which can occur in *day* as well as night operation, although typically to a lesser degree), the resulting corrections serve to improve the performance of the overall system in both day *and* night operation, as shown by the experiments in Section 4.  Indeed, those results show that the proposed approach provides a single architecture and training procedure that outperforms *all* tested baselines across *all* combinations of day and nighttime training and test sequences.\n\nThe paper thus combines a principled design approach with a clear and convincing experimental demonstration of the method’s advantages over prior art.\n\nI did not find any significant weaknesses in the paper (the method of course has its limitations, but in my view these are adequately addressed by the paper itself).\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend accepting this paper on the basis that it addresses an important problem in monocular depth and ego-motion estimation (extending these methods to operate reliably at night), proposes a thoughtfully-designed and well-justified technical approach, and provides clear and convincing experimental evidence of the method’s advantages over prior work, in both day *and* nighttime operation.\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": " Given that this paper is in my opinion already quite strong, I have only a few (minor) comments: \n\n* While it is stated in-line earlier in the paper, I think it might be good to include the definition of Vt(u) in the statement of the full pipeline in equation (4) for easy reference.\n\n* Similarly, it would be good to explicitly state in the paper what are the performance measures reported in Tables I and II.  (RMSE and log RMSE are standard statistical measures, but “Abs. Rel.”, “Sq. Rel”, and the various delta measures are I think less so.  (Also, what is the significance of 1.25, 1.25^2, and 1.25^3 as thresholds?)\n\n* It would also be good to mention how the weighting parameters lambda_r and lambda_g appearing in the loss in equation (5) are chosen.\n"}}, {"review_id": "F3651tqYqSl", "reviewer": "Reviewer_JhM1", "summary": "This paper proposes three techniques to improve self-supervised monocular depth estimation for both day and nighttime images: (i) per-pixel neural intensity transformation for lighting change compensation, (ii) per-pixel residual flow map motion compensation, and (iii) image denoising for robustness improvement. Several constructive insights are introduced in the paper to address the problem of performance degradation in scenarios when the assumption of photometric losses is violated. Experimental results and ablation studies are performed on the Oxford RobotCar dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Improving the depth estimation accuracy of nighttime images is an important problem, and the authors provide three good ideas. This paper presents its novel contributions in a coherent and clear manner; the methodological details for addressing the temporal illumination consistency, static scene, and noise-free assumptions make sense. The experimental results seem convincing as well.\n\nHowever, there are several issues and weakness in this paper, which are listed below: \n\n    * The authors lack elaboration on some technical details. For example, \"lighting change decoder\" has been mentioned in Line 148 but no corresponding network structure is provided. \n\n    * In section 3.4, the authors claimed the denoising model used solely at training time is better than used at both training time and test time due to the better robustness. However, no experimental evidence is shown in the paper to back this claim. \n\n    * Insights for practical implementation of the proposed methods for real-time robot vision or robot learning - are missing. At-least some on-device feasibility analyses / computational reasoning should have been included. \n\n    * This paper lacks a detailed description of the tables and figures that demonstrate the experiment results. For example, “NIT“ in Table 2 is not clearly defined, and the experimental results of Figure 4 are not well explained. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper makes some good contributions toward addressing photometric loss assumptions in low-light conditions and night-time images. Although their practical implementation for real-time robot vision are not straight-forward, but the proposed concepts can be useful to the community. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Authors are encouraged to address the few issues mentioned above to improve clarity and readability of this paper. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "586qD8aQRD", "reviewer": "Reviewer_iBSx", "summary": "The paper introduced an improved network for depth estimation, where the photometric loss is intensified for varying illumination conditions. Specifically, the network is based on an encoder-decoder DepthNet for depth prediction and a MotionNet for pose estimation. The authors proposed three modules to improve the photometric loss for robustness against illumination change (e.g., day to night) and moving objects. They augment the network to predict the contrast and brightness parameters for each pixel to mitigate the lighting change, and predict residual flows to detect moving objects; additionally, they use an image denoising network to remove noise from the images. The evaluation shows the improvements on depth prediction.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The paper is well-written, and the method is clearly stated and technically sound. The comparison against existing work and the ablation study clearly shows the improvements with the intensified photometric loss.\n\nWeakness\n- For novelty, it is an incremental work modified from existing methods. Nevertheless, the performance is good.\n- The reviewer thought the proposed method is also learning-based visual odometry instead of just predicting depth since the ego-motion is also estimated. However, the evaluation is purely based on depth. The reviewer is also interested to see the evaluation of ego-motion.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "It is a good system work for depth estimation in challenging conditions, the evaluation clearly shows the improvement. But the reviewer is not sure about the novelty of the method.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- Section 3.5 is not very intuitive. It might be easy to follow if the equations are replaced with descriptions.\n- The metrics used in the tables need explanation and unit.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)