Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper addresses the problem of depth estimation using a monocular camera and a 2D LiDAR sensors. The paper proposes an anytime method based on a network architecture that is composed of two dedicated encoders map to extract features from the input of each sensor, a unit to fuse such features, and a decoder to get the final depth map. The proposed method accounts for time/energy budget available to potentially stop the inference and get the corresponding result. Experimental results obtained on public datasets are shown to validate the proposed method.
strengths_and_weaknesses: Based on existing networks, the paper presents an interesting idea about stopping the inference and explains the motivation behind the choices that resulted in a much smaller network compared to other methods, displaying comparable or better performance in some cases. The paper is also generally well written.

There are two main comments:
- the paper could provide better examples where this stopping is useful for energy purpose: a self-driving car or any moving robot in general typically consumes much more energy from the actuators rather than from the computer. It would be useful to quantify the energy saving.
- the budget model (constant) is quite simplistic and might result in prematurely stopping the inference, as the CPU load might be different at different times. A discussion on a more realistic model would be useful.
summary_of_recommendation: While the paper could provide a better motivation and discuss a more realistic budget model, the paper presents an interesting anytime method for depth estimation targeted for low-powered systems.


issues: The issues that could be addressed to revise the paper are reported in the "Strengths And Weaknesses" textbox.

Other specific comments:
- in practice, it might be difficult to get the line of points at the center of the ground truth depth map, unless the camera and LiDAR are colocated. Thus, taking the line above or below the center would be more realistic.
- Besides just reporting the data, it will be useful to comment on when other methods perform better than the proposed one, e.g., in Table 3. 

Minor typos: 
- "equiped" -> "equipped"
- "an uniform" -> "a uniform" (because of the consonantal sound)
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a depth estimation method with 2D lidar measurements and monocular RGB images as inputs, which could perform robustly with sparse and highly non-uniform depth measurements, and give reliable estimations when computation budget is limited.
strengths_and_weaknesses: The experimental results on NYU dataset clearly show the advantages of proposed method when working with very sparse depth measurements and low time budget over other state-of-the-art methods,  thus it could establish a new baseline and be a good reference work for relevant research in the future.

This work aims at training a model that provides robust predictions in anytime, hence samples time budget uniformly. I think it would also be interesting to study the actual distribution of interruption time in real applications and applies it to the sampling process, then the model may achieve better expected accuracy.

summary_of_recommendation: This work targets a practical problem and provides a solution with better performance than other existing methods within the working situation, hence I think it has a high technical value. In addition, the proposed method has a clear and simple framework, thus it’s likely to be embedded to other applications conveniently.
issues: 1. I have a question about the first way to optimize the objective in section 4.1: As the budget is sampled from the uniform distribution within (0, Bmax), the model should not be able to control it. But it’s said to minimize the budget here, then how does it work? I guess that the value of Bmax gradually decreases during training so that the model is trained to keep a similar performance within lower expected budget. Am I correct? 

2. Just a suggestion here: The formula of the loss function used in the optimization objective (equation (1)) is not mentioned. I think it would be helpful to the reader if you could briefly mention about it.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work proposes an approach for anytime monocular depth estimation on mobile devices when a structured sparse set of measurements in the form of a 2D lidar is available. The authors propose to encode LiDAR scanline measurements using a neural network and splat them to image features that are encoded using a light-weight image network. 
strengths_and_weaknesses: Strengths:

- Anytime estimation in this setting (e.g. with additional external measurements from LiDAR) is to the best of my knowledge novel. 

- Important and undeserved topic. There are only few works in this general direction despite its obvious usefulness for robotics applications.

- Good results in comparison to other real-time approaches.

Weaknesses:

Clarity is an issue throughout the manuscript:

- Section 3.2 states a relatively simple concepts in a bit heavy-handed terms because of the extensive use of mathematical notation. I'd recommend to make this a bit more accessible by removing jargon and explaining with words. 

- Section 4.1: similar consideration hold about overly complicated notation, but here I'm not sure if I completely understand the motivation here. For point one: how can you guarantee even in expectation the inequality if the losses are only required to be equal? What happens if the loss for \hat f_i is actully slightly larger on |\hat P_i} is only slightly smaller that |P_i|?

- Paper organization: Section 4.2 talks about network architecture followed by an own section 4.3 that is actually titled "network architecture" but then actually goes on to talk about computational properties. This is confusing. It is not clear what "encoder optimization" means. What is optimization referring to here? The ECU is used as a component and how it qualitatively functions is mentioned already in Section 4.2, but it only introduced in 4.3.2. 

- Section 4.3.2 is very hard to understand. What alignment is necessary here? Why is this necessary? How big are the regions? What is the compensation value? It seems that this is just about upsampling the scan line to the full image by extrapolating the scan line in vertical dimension. Why is this so complicated?

- Is a uniform distribution between 0 and B_max well justified? In practice it seems that this would more like by an interval [B_min, B_max] or even a normal distribution.
summary_of_recommendation: I really like the direction that this paper is taking. Unfortunately, the paper in its current form is extremely hard to read and in its current form not fit for publication. Technical aspects are consequently hard to evaluate in detail, but I believe that the paper in principle is on a good direction. I'm willing to significantly improve my score if the revision appropriately improves clarity. 

-- Post rebuttal --

The authors answered my questions and revised the manuscript. I'm consequently raising my score.
issues: Minor issues

- l. 121 abbreviation "r.v." is never introduced.

- The abbreviation PC is used in almost every sentence in some paragraphs which make these parts dull to read.

Missing citation: 

[A] also proposes an efficient depth estimation network that outperforms [18].

[A] Nekrasov et al. Real-time joint semantic segmentation and depth estimation using asymmetric annotation. ICRA 2019
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents  RGB-D based, computationally efficient, encoder-decoder style depth estimation network with a focus on low power, frequently halting applications.   Unlike other RGB-D fusion architectures, to mimic the limitations existed on small devices, authors use RGB and 2D point cloud as model's input instead of 3D point clouds, and propose an Error Correction Unit, ECU, to algin 2D point cloud and RGB image features. ECU achieves that by extracting point cloud's neighbors in image features and downsizing it to match point cloud feature dimension, and the combined feature is upscaled back to image feature dimension.  Additionally, they incorporate five depth estimation outputs along the computation pipeline in order to achieve anytime depth estimation, where predictions with different levels of detail are available regardless of the halting point of the network.  Authors propose a new metric, anytime loss, which is a weighted sum of loss for each depth output based on everyone's computational time. 
strengths_and_weaknesses: Strengths:
1. The idea of anytime depth estimator is novel and useful practically for time-sensitive and low power edge devices. 
2. Error correction unit seems like an interesting approach to fuse image and 2D LiDAR features with different dimensions, existing methods commonly use interpolations to expand LiDAR features to image dimension.
3. Comparing results of common models and authors' model on small devices like Jetson and Raspberry Pi with limited time and energy budget is useful for future applications on such edge devices. As many publications focus on SOTA performance on powerful devices, obtaining results on weak devices can guide the community to pay attention to efficiency and ease of deployment of their networks.

Weaknesses:
1. Since all but the first nearest interpolation predictions require the encoder section of the network to be completed, and the first prediction is significantly worse than others, this paper's claim about anytime estimations still requires at least half of the total computation time to yield useful predictions. (In Table 1 path1 takes 6.4ms whereas path4 takes 12.2ms). Would be nice to alter the network architecture to allow it to bypass some layers of encoders to achieve a lower minimum time requirement.
2. Another question I have about architecture is that although authors claim that the network is based on U-net and uses first three conv layers of MobileNetV1 as encoder. Yet, MobileNectV1's first three layers only reduce image dimension by half, and as 1x1 conv layer is used to generate full-size depth estimation. I am not sure if the network downsamples and upsamples features similar to U-net. 
3. I think this paper doesn't demonstrate the effectiveness of ECU module, as from the results, Parse, which uses 2D LiDAR by interpolation, performs similarly compared to the proposed network. Would be nice to have a comparison between proposed network with and without ECU module.
4. I think "FusionMapping: Learning Depth Prediction with Monocular Images and 2D Laser Scans" should be included as baseline as it shares a similar architecture with the proposed network and uses 2D LiDAR as input as well.
5. Finally, I have a different opinion about the proposed anytime estimator approach to handle limited time and energy budget for edge devices. The paper considers the halting caused by limited budgets to be a uniformly distributed random variable from 0 to B_mas, and designs the model to give prediction whenever halting happens. However, I would argue that the capturing and processing time for RGBD image can be considered as a constant. Thus, instead of designing a model to perform well under different budgets, constructing a model to fit the given budget seems to be a better option.


summary_of_recommendation: Overall, this paper shows a novel approach for low power edge device machine learning by handling limited time budgets by anytime depth estimation. The proposed architecture is similar to other RGBD depth estimators with encoder-decoder backbone and point cloud feature fusion, yet the authors introduce multiple branches of prediction between decoder layers to obtain depth estimation with different levels of qualities. Also, a novel anytime loss is introduced to weigh each output's loss by its computational time. Another novel part is the error correction module for RGB and point cloud feature fusion. The paper demonstrates the network's low latency and good performance comparable to existing methods on low power Jetson and Raspberry Pi. However, the newly introduced ECU layer lacks experimental results to demonstrate its effectiveness. And of the claim of anytime prediction, the model design cannot give outputs if halt happens during encoding. Additionally, random halting doesn't match with the situation on low power devices, instead, a fixed time budget seems to be common, which wouldn't have random halting problem and allows a purposely designed model to fully utilize the time budget. 
issues: 1. Compare the proposed network with and without ECU to demonstrate its effectiveness.
2. add FusionMapping as a baseline 
3. try have halt in encoder step and provide the anytime loss for that.
4. clarify the architecture and if not doing downsampling and upsampling, justify the design choice.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper presents a mono depth completion method based on a single 2D LIDAR scan. The whole model is designed to have a limited computational budget and to exit the inference at different stages depending on the currently available power or computational budget. It shows outstanding results even though the amount of weights and the amount of flops are drastically reduced. 
strengths_and_weaknesses: Strengths:
* Nice and clear presentation of the approach
* Well written paper
* In depth evaluation of the method and comparison on two different relevant dataset
* Introduction on an error-correction unit (ECU), which fuses the 2D features from a mobilenet with the features extracted from a 1D conv-net, by extracting the local neighborhood in the spatial image feature domain and concatenating it with the results from the 1D LIDAR scan features. 
* Four way exit approach, which allows to get results after only half the time of execution of the full pipeline
* Impressive results on NYU Depth v2, where after 6.4ms the method performs already 5% better than FastDepth [18], which needs in comparison 17.1ms

Weaknesses:
* Figure 5 should include the 33 and 50 ms mark, to better compare it with Table 2. 
* Line 267: should better explain that the Raspberry Pi 4 takes 59 ms to get to the first path, which means that is not fast enough for a fair comparison here

summary_of_recommendation: The paper was well written and fun to read. I can strongly recommend accepting this paper to CORL. I have a few pointers, which should be improved in a final version, but nothing which should prevent this paper from getting accepted.


issues: * Figure 5 should include the 33 and 50 ms mark, to better compare it with Table 2. 
* Line 267: should better explain that the Raspberry Pi 4 takes 59 ms to get to the first path, which means that is not fast enough for a fair comparison here
review_title: null

Meta-review: The paper presents a mono depth completion method based on a single 2D LIDAR scan.
The paper proposes an anytime method based on a network architecture that is composed of two dedicated encoders map to extract features from the input of each sensor, a unit to fuse such features, and a decoder to get the final depth map. Overall, this paper shows a novel approach for low power edge device machine learning by handling limited time budgets by anytime depth estimation.

All reviewers but one are positive and find the paper addressing an important question and well written/executed with interesting results. 
- Clarity on how the budget is used: constant or adaptive? If latter how, and what are the implications of the assumptions on uniform sampling vs say exponential or gaussian distributions. 
- Choice and justification for architecture (as pointed out by Reviewer HGbf)

Overall the rebuttal was very useful in clarifying these questions. The authors should ensure that most points from the rebuttal are included in the main paper and if not then should be added to the appendix as FAQs. This will improve comprehension and avoid similar confusion among the readers. 
Decision: Accept (Poster)