Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a pseudo-Lidar framework from sparse Lidar points and an RGB camera. A two-stage network is utilized which preprocesses the sparse Lidar points and fuses them with RGB information to obtain a coarse depth prediction and then refines this. During training, multiple time frames are used to obtain additional losses to compensate the lack of supervised depth data.
strengths_and_weaknesses: Although, categorically monocular 3D object detection includes pseudo-lidar based methods, once the depth information is included (sparse or dense), it is not monocular camera estimation anymore, it includes at least two sensors for prediction. This is a repeated mistake in many papers. 


If multiple frames can be used, then can't their sparse depth correspondences aggregated over time to produce a denser point cloud which could help depth correction as well?

For depth prediction, did you submit your results to original KITTI page?

What is the difference between your reported results for PseudoLidar++ and KITTI page, can you explain briefly in the paper? (For all tables.)

Minor points:

Our experiments demonstrates-> Our experiments demonstrate
summary_of_recommendation: The authors have compared a number of methods. Their own proposal, which is a combination of multiple networks, is reported to achieve a high accuracy. Although I could not check it in the original submission page of the datasets. The codes are also provided, which I appreciate. However, I did not have time to check the codes.
issues: - more discussion on the reason of network module selection.
- explanation of the difference between the dataset benchmark evaluation and your reported results.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes an architecture and self-supervised learning approach for dense monocular depth estimation when a very sparse set of accurate depth readings are available from a LiDAR. The proposed architecture is a multi-stag architecture that consists of a network that receives input images and densified sparse LiDAR readings and produces and inital depth map. The initial depth maps is refined using a refinement network that essentially distills the GDC depth correction from PseudoLIDAR++ into an efficient network. Training is done using self-supervision based on photometric cues and the output of the offline depth correction module. Experiments are shown on KITTI, including results on a 3D object detection downstream tasks.
strengths_and_weaknesses: Strengths:

- Seemingly good results at very high frame-rates. However, I do have some questions about the results (see below).
- I particularly like that experiments on a down-stream tasks are shown. The improvement shown there is impressive.
- Reasonable architecture construction that consolidates the individual stages that have been layed out in Pseudo-LiDAR++ into a single efficient end-to-end pipeline. 


Weaknesses: 

- The comparisons in Table 1 are not quite clear to me. For example the BTS is shown with \delta_1 of 0.904, whereas the original paper shows results somewhere in the range of 0.95 (depending on the employed backbone) - this would actually outperform the proposed work without any LiDAR points required. Similarly DORN in the original paper also shows values of up to 0.93, but the table here only lists it with 0.897. I'd be grateful if the authors could clarify the discrepancy here.

- This is unfortunately shared by many works in this space: the approach is only evaluated on KITTI. No other datasets are shown. Self-supervised approaches promise, in principle, that they can be deployed anywhere (in this work of course conditioned on the availability of LiDAR measurements). It would be great to go beyond KITTI and actually show off the flexibility of this approach. This could be done by showing results on transfer or fine-tuning on other datasets (for example Apolloscape) rather than squeezing every last drop of performance out of KITTI.

Other: 

- Is the pose network really required? It seems that temporally consecutive LiDAR point clouds could be aligned using standard geometric approaches to get the pose. This procedure would be guaranteed to be generalization, unlike a pose network which are known to be hard to generalize (have you tried playing a sequence backward? - many pose networks fail to generalize even to this simple task)


summary_of_recommendation: This work provides a technically well executed pipeline that seems to show good results, especially on the downstream task, which I believe deserve to be seen. However, I do have questions about the evaluation which I hope the authors can resolve.

-- Post rebuttal --
The authors resolved my major questions. What remains is the somewhat evaluation only on KITTI. I do believe however that this shouldn't be a reason for rejection. I thus keep my initial rating.
issues: Minor issues:

- l. 92: Previous text refers to [1] as Pseudo-LiDAR++, but here suddenly as PL++ without an introduction of the abbreviation.

- Table 1: "excepctionall time-consume" - broken sentence.

- Table 2 is hard to read because of small fonts in combination with misaligned table rulers.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper predicts depth images based on monocular RGB images and sparse LIDAR data, by fusing both in there so called DepthNet, which is afterwards improved by RefineNet architecture, which uses the result, the features from the sparse LIDAR data and the sparse LIDAR data itself. They show in a extensive evaluation section superior results to prior state-of-the-art methods in different categories. Furthermore, they show how each single change affects the final result in an depth ablation study. 
strengths_and_weaknesses: Strengths:
* the paper is well written and explains all important concepts in depth
* present a novel fusion mechanism by reusing the features from the first network in the second to refine the depth image
* they claim to be the new state of the art method for depth prediction
* Extra point for including a point cloud result, as no one can properly judge how well a depth completion method works by looking at a colored image

Weaknesses:
* The related work section, while citing many papers lacks a deeper comparison to these prior methods, to better clarify the difference between those works and this new work
* Pseudo Dense Representation: Here they explain how they generate a dense depth map based soley on the sparse LIDAR data. However, they don't answer what happens if more than one sparse data point is close to an evaluated position, this should be added. Is the last point used or is interpolated between the two?
* Table 1 contains a lot of comparisons to methods, which use different input data. 22 rows use only monocular or stereo data and don't use any sparse LIDAR data as an input, which makes the comparison a bit unfair.
* I am furthermore confused by the claim that they beat state of the art, if I check out the leaderboard and try to compare it. I already have the problem that they don't provide the results on the test set but on the validation set, for example they report for "Sparse to dense [30]" an iRMSE of 4.28, while in the leaderboard: http://www.cvlibs.net/datasets/kitti/eval_depth.php?benchmark=depth_completion it says 2.80. I don't understand why they didn't evaluate the test data online, to make it easier to compare it existing methods. 
* Furthermore, do I struggle with the results report for RSME, as they seem much closer to iRSME values than to actual RSME values for KITTI. 
* Lastly, why are the results for the stereo case of Pseudo Lidar++ not reported, I agree that it makes it more fair to only use monocular to make the comparison more fair. That does not mean though that one should not report the original values, because now I can't judge if this change complete breaks the performance of this algorithm and then gives you edge or if you would have beaten it even with stereo? For me to accept this paper I need to know these values to access if changing Pseudo Lidar++ to monocular is something which the architecture can handle or if you broke Pseudo Lidar++ on purpose to give your method an advantage. As we can only judge other methods as they published them not as we want them to be.
summary_of_recommendation: I am mostly basing my review on my doubts about the evaluation. If the authors can prove or explain the choices they made in their evaluation then I can gladly switch my recommendation to strong accept. But, at the moment I feel like that the evaluation was designed in a way to make it harder to compare it to the leaderboard and weaken on purpose the other approaches to boost their own method. 

Based on that I can not recommend accepting this paper in its current form to CORL.

After rebuttal:
The authors answered most of my problems with the evaluation and henceforth I have no objection anymore against accepting this paper to CORL. They adjusted the evaluation and positioned themselves better against Pseudo LiDAR++. They also explained why an evaluation on the validation set is justified and extended the related work.

issues: * Results for Pseudo LIDAR++ with stereo have to be included
* The depth completion and prediction methods should be evaluated on the test set not the validation set and furthermore the values should be in the same format as in the online leader board, to make it easier for everyone to understand if the claim about state of the art performance is correct
* The related work section should contain more explanation on how this method compares to existing methods and not just list them.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method for fusing sparse lidar point clouds with RGB images to obtain accurate dense depth predictions. Key to the performance of the approach is the use of pseudo-dense representations of the sparse lidar point cloud which assist the convolutional network in effectively fusing the data. The results show that the approach outperforms a range of current methods in terms of depth completion, monocular depth prediction and stereo depth prediction.

strengths_and_weaknesses: **Strengths**
- The paper is very well written and polished. The figures are nicely presented and help to explain the approach. The paper seems to be mature and is of publication quality should it be accepted.
- I think the approach is very interesting and well thought out. What I particularly like about it is that multiple sources, i.e. sparse depth, stereo supervision and monocular training are all combined elegantly into one framework. Many approaches can only train with monocular supervision which is quite limiting. 
- The experimental evaluation is extensive and confirms the claims made in the paper. A good ablation study is done showing how each component contributes to the performance. The method is compared to a multitude of existing approaches. The proposed method performs very well compared to these. Including some other tasks such as mono-only and stereo prediction in the evaluation is useful as it allows the reader to place the method holistically (not just focussing on depth completion). This is definitely very useful as a comparison when making decisions for a real system. (However, see weakness below as I think this has been overdone a bit).
**Weakness**
- While the evaluation is quite extensive, I think the proposed approach is more of a depth completion approach for which there are a number of benchmarks (like the KITTI online depth completion benchmark) and many existing methods. However, the paper focuses extensively on monodepth methods in the comparison. This seems a bit misleading to me as it is obvious that mono approaches will perform worse. Maybe using this space to show the performance in other environments would be better. Also, why has the approach not been evaluated on the online benchmark?
- The approach has only been evaluated on KITTI where there is lots of repeated and regular structure. How does the method work in a general robotic setup such as would be the case for a home robot in an indoor environment? Or a robot in an outdoor field or sidewalk?
The improvement gained by the approach on the depth completion evaluation is quite small i.e. the existing methods already perform quite well.
- There is no comparison between the capacity of the proposed network and those in the literature. Does the method use significantly more layers or weights?

summary_of_recommendation: The paper itself is well written and I like both the manner in which the sparse lidar is effectively fused with the RGB and the natural manner in which multiple sources of supervision can be used. The main weakness though is that this only results in an incremental improvement on the (many) existing methods for depth completion. I do not see what sets this paper apart.
issues: I would like the authors to address the following points:
- How does the method perform in more challenging indoor scenes?
- I think the runtime results should be moved into the main paper as this is quite important for the reader to know
review_title: null

Meta-review: The paper proposes an approach for self-supervised depth estimation from sparse LiDAR scans. The reviewers find the paper to be generally well written and well structured. The reviewers appreciate the experiments on downstream tasks and the good results. Most of the major concerns raised by the reviewers have been addressed in the rebuttal, except for the benchmarking on the KITTI test set. For this the authors have provided the reasoning on why its unfair to compare to methods on the leaderboard. I agree to an extent but it would still be nice to see how well the method performs, getting a lower metric value on the leaderboard will not diminish the contributions of the paper. Nevertheless, I recommend acceptance and I encourage the authors to still benchmark their approach on the KITTI depth completion or depth prediction test set.
Decision: Accept (Poster)