Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors introduce the problem of selecting a subset of lines from a lidar scan that maximizes the accuracy of the depth map one obtains when completing the lidar line selection to compare against the original complete depth map. A solution to this line subset finding problem is presented which selects scan lines one by one based on their marginal contribution to the current line selection while encouraging spatial spread of the selected scan lines. The proposed method is evaluated on the KITTI dataset in conjunction with a deep-learning-based depth completion network. Various adaptations of the approach, such as various spatial sampling techniques and whether the line selection occurs once for all frames (global) or at every frame (local), are evaluated on the RMSE metric against the ground-truth lidar scans.
strengths_and_weaknesses: # Strengths

The approach is technically sound and explained clearly by introducing the Shapley value and the proposed spatial sampling techniques.

The evaluation of the method on the KITTI benchmark is detailed, with extensive ablations over the spatial sampling strategy and performance results, albeit only the RMSE is considered as accuracy metric.

# Weaknesses

The motivation for the introduced problem seems questionable to me. It is mentioned that "lowering the costs of access to lidar" (l. 280) motivated the proposed method. However, I do not see how it is economical to design new custom lidar hardware based on a non-uniform line pattern that has been found by this method when regular lidars measure equidistant scan lines. Do the authors have any evidence for the feasibility of lowering lidar cost through their approach?
Futhermore, when the line selection is allowed to change at every frame (Shapley local) or even between datasets -- what is the practical benefit/application of this? If the computing time can be reduced through less input data to perception pipelines, is the proposed method fast enough to warrant this additional lidar scan "downsampling" step at every frame? When we focus our attention to downsampling, how does Shapley-local compare to other approaches in this area, such as voxel-based downsampling?

Evaluating the method on a single dataset which has the same lidar placement accross all measurements potentially limits the generality of the proposed method. It would be interesting how the selection of lidar lines is affected by the tilt or height (for example) of the lidar, even on simulated data. Tying it back to the aforementioned practicality problem of designing custom lidar hardware -- is there any reason to assume that the same line selection will work globally under different conditions?

The paper does not discuss the choice of depth completion algorithm. Only in the experiment section is it mentioned that the approach from [5], a self-supervised deep learning method, is used as this crucial component which directly affects the performance of the overall pipeline. It should be explicitly said and better motivated in Sec. 2 that [5] is used as depth completion module. Furthermore, it has to be explained whether the network needs to be retrained for different line selections and what hyperparameters have been used. Do these hyperparameters influence which of the proposed line selection strategies yield the best results? If so, they must be included in the performance evaluations.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: I cannot recommend acceptance of this work in its current form because it is difficult to assess the relevance to robotics of the introduced problem space and the proposed method. The authors should find a more convincing angle to motivate line selection (both global and local) and consider positioning the paper as an approach of downsampling of lidar scans, which is well motivated and has many well established baselines to compare against.

Futher issues to address to improve clarity and overall strength of the paper are given below.
issues: # Major Issues

It seems that the appendix is missing for this submission. In line 164 a "supplement" is mentioned to describe the criteria of distribution fairness. In line 245 the authors again refer to an appendix.

Performance numbers should be provided for the given experiments. In particular for the local approach where the line selection happens at every frame, it is important to know whether the approach can run in real time. On that note, how many binary vectors were sampled to evaluate the Shapley values for the line selection candidates?

No limitations of the proposed method have been discussed in the paper.

# Minor Issues

It would be clearer to see how the method compares in Fig. 2 to only show the line diagram in Fig. 2 (right) without the table, and plot the evolution of the RMSE of all the methods.

Eq. 1 seems that it should not have the $\arg$ operator in it.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method for reducing the cost of accessing LiDAR for depth completion task. To achieve the saving, LiDAR lines are ranked with Shapley value. Lines with high Shapley values are considered important and kept while others are discarded.
strengths_and_weaknesses: Strength:
- Method description is clear and easy to follow.
- Modeling the importance of the LiDAR line with Shapley value looks natural and also promising.
- Experiments for accuracy evaluation show the power of this method

Weakness:
- Abstract is hard to read
- The objective of this method is ambiguous. What cost is saved with your method?
  - Energy consumption for LiDAR? In my understanding, to assess the importance of the LiDAR lines, this method requires the data from all LiDAR lines. So no LiDAR lines can be completely shut down.
  - Computation for depth completion? As far as I know, most depth completion methods project the point cloud collected from LiDAR to a 2D plane. So no matter how many LiDAR points there are, the inputs to the network are always 2D planes with the same number of pixels. The cost for inference doesn't change.
- To assess the Shapley value, multiple samplings, inferences, and accuracy evaluations are required. The overhead of this sampling process seems to overwhelm the benefit of line selection, especially in the dynamic problem setting.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations section not present
summary_of_recommendation: The modeling part and the result of this paper look good. Though not supported by current LiDAR, the method proposed in this paper does show the probability of cost saving. Though the objective and overhead are ambiguous in the main text, they are clarified during the rebuttal period. Thus I change my score and recommend accepting this paper.
issues: The objective and the overhead mentioned in the weakness section.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposed a solution to select the most effective LiDAR lines for the depth completion task. Major contributions are as follows:

(1) The author proposed to leverage the Shapley value for LiDAR lines to evaluate the marginal contribution to the overall depth completion output. 
(2) A spatially-aware Shapley value (SaS) scheme is proposed to take into account the spatial configuration of the line selection.
(3) The proposed method has two basic variants of the approach, the global version which select lines at a dataset level and the local version which selects lines for each image independently.
strengths_and_weaknesses: **Strengths:**
* This paper proposed a novel method to approximate the marginal contribution of a LiDAR line to the depth completion task, which could provide valuable information for the sparse LiDAR design.
**Weaknesses:**
* In my understanding, the proposed method needs pre-training on the dense (64-line) LiDAR data before the line selection, which hinders its application on Sparse-LiDAR-only systems.
* The computation cost analysis is missing.
* It would be better if the line selection could be efficiently predicted only with the RGB image input, to guide the LiDAR design or selection.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations section not present
summary_of_recommendation: The LiDAR line selection could greatly affect the depth completion performance because different lines contribute a different amount of importance. The problem author tried to solve is significant. Although the method and presentation are not perfect, it is still inspiring.

Post-rebuttal: the proposed method seems not practical for current rotation-based LiDARs, but it is still inspiring and may be possible to guide the design of future solid-state LiDARs which could generate variational scanning patterns. I would keep my rating of "weakly accept"
issues: **Major Issues:**
* The limitation section is missing.
* Analysis of computation cost is needed.
**Minor Issues:**
* The author could detailly state how the proposed method could contribute to the real-world application. If the line selection requires pre-training or sampling on dense LiDAR data, the value of the proposed method will need further justification.
* There is more SOTA depth completion methods that could be discussed in the Related Works section like:

[1] X. Cheng, P. Wang, C. Guan, and R. Yang. CSPN++: learning context and resource aware convolutional
spatial propagation networks for depth completion. AAAI Press.

[2] Choi, Jaehoon, et al. "Selfdeco: Self-supervised monocular depth completion in challenging indoor environments." 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.

[3] Feng, Ziyue, et al. "Advancing self-supervised monocular depth learning with sparse liDAR." Conference on Robot Learning. PMLR, 2022.

[4] Yoon, Sungho, and Ayoung Kim. "Balanced depth completion between dense depth inference and sparse range measurements via kiss-gp." 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.
review_title: null

Meta-review: metareview: Main strengths:
1. The paper is well written and the technical ideas are clear.
2. Experiments show that the method is effective for LiDAR line selection, but several cases/baselines/metrics of interest not tested (see below).

Main weaknesses:
1. Practical value of the method/target problem is not sufficiently justified.
2. Related to the above, the experiments are lacking thorough evaluation of the method (e.g., overheads) in the context of practical LiDAR systems.

After the rebuttal and discussions, the reviewers have largely kept their original ratings. There is a sense that the work is interesting enough to be accepted so as to spur future research on the topic, but the true usefulness of the the idea in the context practical LiDAR systems need to be further investigated.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)