Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents three contributions to improve self-supervised depth and ego-motion estimation:

1. a novel per-pixel neural intensity transformation network to compensate for lighting changes; 
2. a residual flow map for each pixel to correct the reprojection correspondences;
3. Using a Neighbour2Neighbour algorithm to denoise input images
strengths_and_weaknesses: Strengths:
* The paper is well written and easy to follow.
* The lighting change decoder idea seems interesting and useful.
* The experiments are conducted on the public dataset and tested extensively at different combinations of day and night, with a good separation of training and testing routes.
* The result shows a consistent improvement compared to baseline methods.


Weaknesses:
* In general, the novelty in this paper is somewhat limited. It was not the first time that motion compensation and image denoising on the inputs are proposed for depth estimation and ego-motion learning.
* There are a few issues that could be treated as weaknesses. However, they may also be able to be addressed in the revision by the authors. Please take a look at the issues I listed below. 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper may have a few limited theoretic novelties, but it shows well-conducted experiments and improvements compared to baseline methods. The paper can be improved if the mentioned issues are addressed.
issues: 1. Generalisation: The method is trained and tested in the Oxford RobotCar dataset, with a good separation of training and testing routes. However, I am still a bit curious about whether the system trained in the Oxford Robotcar dataset can work well in other environments, day and night.
2. The assumption of constant illumination is an issue for photometric loss. Some existing works address this issue as well, but instead of using a motion network directly to regress the relative transformation, they formulate it as an optimisation problem. I believe they are also worth being mentioned in the related work section:
    1. Clement, Lee, and Jonathan Kelly. "How to train a cat: Learning canonical appearance transformations for direct visual localization under illumination change." IEEE Robotics and Automation Letters 3.3 (2018): 2447-2454.
    2. Bloesch, Michael, et al. "Learning meshes for dense visual slam." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
    3. Xu, Binbin, Andrew J. Davison, and Stefan Leutenegger. "Deep probabilistic feature-metric tracking." IEEE Robotics and Automation Letters 6.1 (2020): 223-230.
    4. Gridseth, Mona, and Timothy D. Barfoot. "Keeping an Eye on Things: Deep Learned Features for Long-Term Visual Localization." IEEE Robotics and Automation Letters 7.2 (2021): 1016-1023.
3. The idea of capturing the per-pixel changes in contrast (scale) and brightness (shift) looks interesting. I appreciate the theortical analysis in the supporting material to prove this assumption. However, it looks to me that there are too many simplifications involved to derive from the Phong illumination model to the proposed linear combination of scale and shift lighting change equation. In particular, the simplification of removing the reflection term looks a bit suspicious. I think a big portion of the lighting changes that occur at night is the reflection on roads and surrounding vehicles.
4. I am wondering why the proposed method does not utilise geometric consistency. The depth image $D_{t+1}$ from $I_{t+1}$ is not used in the pipeline.
4. The limitation section is a bit too general.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper describes an approach to the self-supervised training of deep networks for joint depth and ego-motion estimation that are capable of accurate all-day (i.e. day *and* night) prediction.  The main technical contribution is to identify three assumptions underpinning the use of the photometric loss for self-supervision that are significantly violated during nighttime operation (temporal illumination consistency, static scenes, and absence of noise and occlusions), and then show how to modify standard architectures for monocular depth and ego-motion estimation to correct for these sources of error.  The result is a single network architecture that can be trained and deployed on any combination of day and nighttime image sequences.  The experimental results provide convincing evidence that the proposed system significantly outperforms comparable baselines on both day and nighttime applications, including prior works specifically designed for nighttime use.
strengths_and_weaknesses: In my opinion the paper’s greatest strength is that the technical approach is explicitly grounded in the physics and geometry of image formation.  

For example, the paper motivates the development of the lighting change compensation module by considering the validity of the “temporal illumination consistency” assumption in daytime operation (where most of the ambient illumination comes from the sun, which is well-approximated as a static source) versus nighttime operation (where much of the ambient illumination comes from traffic lights, streetlamps, and headlights of other vehicles, which are bright point sources that move over time relative to a moving camera).  The proposed illumination compensation method is a simple yet effective linear model based upon a predicted *contrast* map (which is intended to capture *global* changes in illumination caused by motion of the vehicle towards / away from ambient sources such as streetlights) and a predicted *brightness* map (which accounts for the presence of potentially moving *point* sources, such as vehicle headlights).  Similar analyses lead to the motion correction approach (designed to account for motion blur and / or the motion of objects in the scene, such as vehicles) and the denoising approach used to pre-process the training images.

The design of the proposed approach is thus well-justified, physically grounded, and readily interpretable, all of which are highly desirable properties.  

Furthermore, because the approach is based upon accounting for violations of the assumptions underpinning the photometric loss (which can occur in *day* as well as night operation, although typically to a lesser degree), the resulting corrections serve to improve the performance of the overall system in both day *and* night operation, as shown by the experiments in Section 4.  Indeed, those results show that the proposed approach provides a single architecture and training procedure that outperforms *all* tested baselines across *all* combinations of day and nighttime training and test sequences.

The paper thus combines a principled design approach with a clear and convincing experimental demonstration of the method’s advantages over prior art.

I did not find any significant weaknesses in the paper (the method of course has its limitations, but in my view these are adequately addressed by the paper itself).



robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend accepting this paper on the basis that it addresses an important problem in monocular depth and ego-motion estimation (extending these methods to operate reliably at night), proposes a thoughtfully-designed and well-justified technical approach, and provides clear and convincing experimental evidence of the method’s advantages over prior work, in both day *and* nighttime operation.

issues:  Given that this paper is in my opinion already quite strong, I have only a few (minor) comments: 

* While it is stated in-line earlier in the paper, I think it might be good to include the definition of Vt(u) in the statement of the full pipeline in equation (4) for easy reference.

* Similarly, it would be good to explicitly state in the paper what are the performance measures reported in Tables I and II.  (RMSE and log RMSE are standard statistical measures, but “Abs. Rel.”, “Sq. Rel”, and the various delta measures are I think less so.  (Also, what is the significance of 1.25, 1.25^2, and 1.25^3 as thresholds?)

* It would also be good to mention how the weighting parameters lambda_r and lambda_g appearing in the loss in equation (5) are chosen.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes three techniques to improve self-supervised monocular depth estimation for both day and nighttime images: (i) per-pixel neural intensity transformation for lighting change compensation, (ii) per-pixel residual flow map motion compensation, and (iii) image denoising for robustness improvement. Several constructive insights are introduced in the paper to address the problem of performance degradation in scenarios when the assumption of photometric losses is violated. Experimental results and ablation studies are performed on the Oxford RobotCar dataset.
strengths_and_weaknesses: Improving the depth estimation accuracy of nighttime images is an important problem, and the authors provide three good ideas. This paper presents its novel contributions in a coherent and clear manner; the methodological details for addressing the temporal illumination consistency, static scene, and noise-free assumptions make sense. The experimental results seem convincing as well.

However, there are several issues and weakness in this paper, which are listed below: 

    * The authors lack elaboration on some technical details. For example, "lighting change decoder" has been mentioned in Line 148 but no corresponding network structure is provided. 

    * In section 3.4, the authors claimed the denoising model used solely at training time is better than used at both training time and test time due to the better robustness. However, no experimental evidence is shown in the paper to back this claim. 

    * Insights for practical implementation of the proposed methods for real-time robot vision or robot learning - are missing. At-least some on-device feasibility analyses / computational reasoning should have been included. 

    * This paper lacks a detailed description of the tables and figures that demonstrate the experiment results. For example, “NIT“ in Table 2 is not clearly defined, and the experimental results of Figure 4 are not well explained. 
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper makes some good contributions toward addressing photometric loss assumptions in low-light conditions and night-time images. Although their practical implementation for real-time robot vision are not straight-forward, but the proposed concepts can be useful to the community. 
issues: Authors are encouraged to address the few issues mentioned above to improve clarity and readability of this paper.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper introduced an improved network for depth estimation, where the photometric loss is intensified for varying illumination conditions. Specifically, the network is based on an encoder-decoder DepthNet for depth prediction and a MotionNet for pose estimation. The authors proposed three modules to improve the photometric loss for robustness against illumination change (e.g., day to night) and moving objects. They augment the network to predict the contrast and brightness parameters for each pixel to mitigate the lighting change, and predict residual flows to detect moving objects; additionally, they use an image denoising network to remove noise from the images. The evaluation shows the improvements on depth prediction.
strengths_and_weaknesses: Strength
- The paper is well-written, and the method is clearly stated and technically sound. The comparison against existing work and the ablation study clearly shows the improvements with the intensified photometric loss.

Weakness
- For novelty, it is an incremental work modified from existing methods. Nevertheless, the performance is good.
- The reviewer thought the proposed method is also learning-based visual odometry instead of just predicting depth since the ego-motion is also estimated. However, the evaluation is purely based on depth. The reviewer is also interested to see the evaluation of ego-motion.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: It is a good system work for depth estimation in challenging conditions, the evaluation clearly shows the improvement. But the reviewer is not sure about the novelty of the method.
issues: - Section 3.5 is not very intuitive. It might be easy to follow if the equations are replaced with descriptions.
- The metrics used in the tables need explanation and unit.
review_title: null

Meta-review: metareview: The paper proposes a new network for depth and ego-motion estimation. The main contribution is introducing three components in dealing with the photometric losses used in depth estimation methods. Majority of the reviewers agree on the contribution of the paper and acknowledge the novelty of the introduced three components in depth estimation. There are concerns on the generalization of the proposal method and several unclear parts in the paper.

The concerns from the reviewers have been successfully addressed during the rebuttal. The authors are encouraged to revise the final paper accordingly.
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)