Lidar Line Selection with Spatially-Aware Shapley Value for Cost-Efficient Depth Completion
Authors: Kamil Adamczewski
Christos Sakaridis
Vaishakh Patil
Luc Van Gool
Venue: CoRL 2022 Poster (2022)
Keywords: Lidar
Shapley value
feature selection
depth completion

Abstract: Lidar is a vital sensor for estimating the depth of a scene. Typical spinning lidars emit pulses arranged in several horizontal lines and the monetary cost of the sensor increases with the number of these lines. In this work, we present the new problem of optimizing the positioning of lidar lines to find the most effective configuration for the depth completion task. We propose a solution to reduce the number of lines while retaining the up-to-the-mark quality of depth completion. Our method consists of two components, (1) line selection based on the marginal contribution of a line computed via the Shapley value and (2) incorporating line position spread to take into account its need to arrive at image-wide depth completion. Spatially-aware Shapley values (SaS) succeed in selecting line subsets that yield a depth accuracy comparable to the full lidar input while using just half of the lines.

Lidar Line Selection with Spatially-Aaare Shapley Value for Cost-Efficient Depth Completion
Kamil Adamczewskil.2 Christos Sakaridis 'Computer Vision Lab, ETH Zürich
Vaishakh Patil
Lue Van Gool 1.3
MPIIIS KKU Leuven
kamil. adamezewskti@gaail.coo {akamil, csakarid, patil, vangool]@vision.. ee ethz. .c
Abstract: Lidar is a vital sensor for estimating the depth of a scene. Typical spinning lidars emit pulses arranged in several horizontal lines and the monetary cost of the sensor increases with the number of these lines. In this work, we present the new problem of optimizing positioning of lidar lines to find the most effective configuration for the depth completion task. We propose solution reduce the number of lines while retaining up-to-the-mart quality of depth completion. Our method consists of two components, (1) line selection based on the marginal contribution of a line computed via the Shapley value and (2) incorporating line position spread to take into account its need to arrive at image- wide depth completion. Spatially-aware Shapley values (SaS) succeed in selecting line subsets that yield depth accuracy comparable to the full lidar input while using just half of the lines.
Keywords: lidar, depth completion, feature selection, Shapley value
Introduction
Lidars have become indispensable for outdoor applications such autonomous cars. They provide highly accurate range information at fairly dense resolution compared to other active sensors such as radars. Moreover, their results are quite independent of the degree to which the surrounding scene is illuminated. The range information from a lidar provides a valuable signal for depth completion, i.e. for the estimation of high-resolution depth map from an input camera image along with the lidar mcasurements. In such a setting, the accuracy of the completed depth map depends highly on the density of the lidar measurements. For the commonly used spinning type of lidar, this density is determined by the number of pulses emitted by the sensor at each azimuth, which corresponds the number of horizontal scanning lines that the measurements form.
The cost of lidar sensors goes up with the number of scanning lines. Hence, increasing the measure- ment density tends to increase the overall cost. Consequently, performing depth completion based on fewer lidar lines is desirable. A naive approach to select a subset of lidar lines is keeping lines at regular, spatial intervals, which is the standard in the industry. But is this an optimal set-up, or does make sense try to build hardware with a custom line set-up? And if so, how can one achieve the latter? These are the main questions that this work attempts to answer.
We argue in this work that equally spaced line selection ignores the non- non-uniformity of the depth profile in real-world scenes. In particular, certain parts the scene contain others, which implies that using more lidar lines for measuring these parts can support more accurate depth map completion. Then, the line selection could greatly affect the depth completion performance because different lines contribute a different amount of importance. This work elaborates on the algorithmic approach how the custom lines can be selected and placed, thus providing the case that lidars with custom lines can be viable option for further academic and industry research.
structures than
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

In this paper, we propose an adaptive non-uniform selection of lidar lines for depth completion. To accomplish this, we utilize the so-called Shapley value [1] from the area of feature selection. The computed Shapley value for each lidar line indicates its marginal contribution to the overall depth completion output. This allows our method to directly evaluate the importance of each line for this task and keep the lines which are deemed most important. limitation of these basic variants is that they treat lines that are far apart. To account for this, we propose spatially-aware Shapley value (SaS) scheme, in which the line selection further takes into account the spatial configuration of the lines by enforcing their selection to exhibit a minimum degree of regularity.
unordered set, even though adjacent lines exhibit higher correlation than lines
Furthermore, we introduce two basic variants of the approach, which select lines either at global dataset level at local image level. In the former case, the scanning lines are fixed for the sensor. The entire line selection process is carried out once and then the identified distribution can be used to build custom non-uniform sensor. In the latter case, the selected lines may differ between any two given images.
We experimentally validate the proposed lidar line selection approach on the KITTI dataset [2] for depth completion. The results demonstrate that SaS substantially outperforms both straightforward baselines and the basic Shapley value variants described above. SaS can maintain a depth error comparable to the error when using the full set of 64 lines, with few as 32 lines. Our findings suggest that accurate depth sensing is attainable even with reduced number of lines, which can make the usage of lidars in mass-proddction sensor suites for autonomous cars more feasible.
2 Related Work
Depth completion from sparse lidar measurements and a single RGB image was addressed in [3], which used single deep regression network to predict depth. This work showed the benefit of using even few lidar samples for the increase of prediction accuracy. An encoder-decoder architecture is proposed in [4] for handling both sparse lidar and dense image data without the need for validity masks for the former. The requirement for semi-deme dense depth annotations is alleviated in the self- eeff-supervised approach of [5]. Surface normals are leveraged as intermediate or additional representations for densifying depth in [6, 7]. A fusion strategy is used in [8] to incorporate the information from the RGB image and correct potential errors in the lidar inputs. Bayesian approach is followed in [9] to assign a posterior distribution to the depth of each pixel, modeling the sparse lidar points via likelihood terms. Non-local neighborhoods are proposed in [10] to iteratively propagate depth values across the image, while graph-based neighborhoods for propagation convolutions. Spatially-varianl convolutional kernels inspired from guided image filtering are used in [13] to adaptively fuse features from the lidar and the camera branch. A cascaded scheme for depth prediction is presented in [14], using two complementary branches to compute the final depth map. A transformer- network is introduced in [15], dynamically exploiting context across camera and lidar. An unsupervised framework for depth completion proposed in [16], which learns to complete depth from sequences of measurements by backprojecting pixels to 3D space and minimizing the photometric reprojection error induced by the predicted depth. Crucially, all aforementioned methods either assume the existence of full lidar scan, typically with 64 scanning lines, which comes from a very expensive sensor, or use random subset of the lidar points. However, efficiently sampling random points from a spinning lidar is not feasible, as the mechanics of the sensor restrict potential subsampling to the level of entire scanning lines. Our method takes this constraint into account and automatically selects a subset of lidar lines by computing the importance of each line for the accuracy of the predicted depth.
utilized in [11, 12] via graph
Feature selection and importance estimation focuses on the effects that representations involved in learned models (such as neural networks) have on the final output of the models. This analysis aims explaining the inner workings of the models. For a comprehensive overview of recent works on interpretable machine learning, we refer the reader to [17] and focus here instance-wis feature explanation, which is related to our work. One line of work in this area consists of caature-additiv

---

methods, such as LIME [18], SHAP [19], and attribution- based methods [20]. These methods estimate the importance of deep features along the channel dimension. LIME makes an assumption of local linearity of the examined model and permutes the features of new samples to weigh them according to their proximity to the model. SHAP is more closely related to our work, as it also uses the Shapley value [1] explain the model by computing the deviation a given data sample exhibits from the global average of each feature. Both these methods output feature importance weight, similarly to work. Another line of work includes feature selection methods, such as 1.2X [21] and INVASE [22]. These methods aim at identifying subset of the original features of the network that yields a similar output to that corresponding to the full set of features. This selection hard, in the sense that the decision whether to keep or neglect each feature not associated with a continuous importance score but rather with a binary variable. This stands in contrast with our Shapley value-based approach, which assigns continuous score to each feature.
3 Problem Formulation
3.1 Definition of Line
In this work, refer interchangeably to line, channel, a lidar line a lidar scan line as defined below. Lidar lines are separate lidar channels (or scan lines) from the lidar point cloud, as provided by the KITTI dataset [2]. The point clouds were generated with a Velodyne HDL-64E lidar sensor. Given point cloud, we denote a point of it by P. = (æ,, Vi. 2;) E R3 We denote the position the sensor by Pa. We adopt the same coordinate system convention as [2]: x-axis (front), y-axis (left) and z-axis (up). If we consider horizontal plane including the sensor, then the elevation angle 0g with respect to this plane for point P, is given by:
(P. Pa); ( P
0, arcsin
We group the resulting angles based on details provided in the Velodyne HDL-64E manual. The lidar has a vertical field of view of 26.9 and its vertical angular resolution is 0.4% on average. Although there are 64 channels, only 42 lidar channels overlap with the camera frustum.
3.2 Line Selection
In our line selection strategy, two modes are distinguished, static (global) and dynamic (local). In the static setting, the set of selected lines is fixed across the entire dataset. In the dynamic setting, a different set of lines can be selected for every frame. The global setting relates to a sensor with fixed position where the pulses always emitted towards the same direction. On the other hand, the dynamic setting allows for changing position / orientation of the transmitters. The limited set of lines is made to change depending on the input scene, i.e. on an image to image basis.
The problem of finding the optimal subset nature and would require the search among (?) subsets. Due this computational complexity, we reformulate the problem and, instead of computing the optimal subset directly, we look for the most advantageous individual lines. Thus, aim create ranking of the individual lidar lines. Then for given ranking and for arbitrary k the top k from the ranking form could be selected as the desired outcome.
lines out of n possible positions exponential in
crucial question then is how we create the ranking the lines. Our approach is to find the lidar line which, when added to an existing set of selected lines, would cause the biggest increase in performance. That is, for an arbitrary subset of already selected lines, we search for the line the marginal contribution of which is the largest. We address this point in Sec. 4 by employing the concept of the Shapley value.
3

---

Table 1: Comparison of terminologies in game theory (left) and formulation of lidar lines rankings (right).
Game theory
Ranking in CNNs
Notation
player
line RMSE
72
characteristic function
coalition
subset of lidar lines set of all the lidar lines
K
grand coalition
N
coalition cost
increase in RMSE after removing some of the lines ((K)
4 Method
We divide this section in three parts. First, present the overview of how the Shapley values (SVs) are computed for the lidar lines. Shapley values are real numbers that quantify the average contribution of line in completing the respective depth map. Secondly, we present the approximation of the Shapley values based on linear regression, which allows to compute the SVs by incorporating sampling. Thirdly, due to the nature of the depth completion problem, we incorporate the concept of space or line spread.
4.1 Game Theoretical Lidar Line Ranking
This section tackles the problem of quantifying the role of a single lidar line in creating depth map. The importance of lidar line described as the improvement the quality of the depth map when we include given lidar line. other words, given a set lines, the contribution of a line the difference between the performance of the existing set of lines and the performance of the of lines plus the line in question. To this end, we employ concept from coalitional game theory, the Shapley value, which precisely quantifies the line's importance its average marginal contribution.
4.1.1 Coalitional Game Theory
Let lidar line be called a player, the set all players N := .o..... N the grand coalition and subset players S A coalition of players. Subsequently, we assess the utility given coalition, i.e., of given subset of lines. To assess quantitatively the performance of group of players, each coalition is assigned real number, which is interpreted coalition receives or incurs collectively. The value of coalition is given by characteristic function (a set function) which assigns a real number to a set of players. A characteristic function v as defined before maps each coalition (subset) K. C N to a real number v(C). Therefore, a coalition game is defined by a tuple (N,v).
payoff cost that
In our case, coalition subset of lines and the characteristic function evaluates the performance. which in our case is simply the root mean square eπтоr (RMSE) for the depth map produced based on the given subset of lines with respect to the ground-trute depth map. While in our case we define this characteristic function as the RMSE, it could be simply replaced by another metric such as mean absolute error (MAF), photometric error, etc.
Up until this point, have defined the payoff given group of lines. There remains the question of how to assess the importance of single line given the information about the payoffss for each subset of lines. To this end, we employ the concept of the Shapley value about the normative payoff of the total reward or cost.
4.2 Shapley Value
The concept introduced by Shapley [1] is division payoff scheme which splits the total payoff into individual payoffs given to each separate player. These individual payoff's then called the Shapley

---

values. The Shapley value of a player i E N is given by
pi(v) = E
((CU(()) KENAE) (K]
(2)
The value (v)) then quantifies the contribution of the i-th player to a target quantity defined by v(N) - v0) that is the output of the characteristic function applied to the grand coalition minus the output when applied to the empty set. The sum over the Shapley values of all actors is equal to this target quantity, v(N) (0) = Ei (v)) In our case, the grand coalition is the set of all the lines and the empty coalition corresponds to the case where no lidar measurements are used. Using the Shapley symmetrization ensures that the contributions
estimated in "fair" way, i.e., according mathematically rigorous division scheme that has been proposed as the only measure that satisfies four normative criteria regarding fair payoff distribution. We describe these criteria in the Appendix.
4.3 Approximation via Weighted Least-Squares Regression
The Shapley value approximation describes the sets as binary vectors, where the vector dimensionality is equal to the total number of players. Each binary vector indicates whether line is present in the respective subset or not. This allows to formulate Eq. 2 as weighted least-squares regression problem. Nevertheless, since the exact Shapley value could only be obtained using exponentially many binary vectors, we resort to sampling. Given the subset IC, we create a binary vector v s.t. [v] N. v(v) v(K) and
f1 vi 1 (0 otherwise.
i KC,
Alternatively, we can also sample the binary vectors directly by assigning 1/2 probability be either
O or 1 to each vector entry or sample the vector based on the probability 1/(x) for subset of length K. Consider then the Shapley values sp((v).... xp to be the weights of the binary vector v. As stated in [23], a formulation in this form allows to obtain the Shapley values as the solution of
I ((K)
min (P RCN
-
Σ p))()) JEK
A(N,K)),
where k(N, are called the Shapley kernel weights which are defined as k(V, K.) (IW1-1) ((())1<))))))) where k<,,,N) is set large number due to the division by O. In practice, the minimization problem in Eq. 3 can then be solved by solving a weighted least-squares regression problem, the solution of which is
= (V
where V is a matrix consisting of the above defined binary vectors, K the Shapley kernel weight matrix, and is vector with the outcomes of the characteristic function applied to the corresponding subset in V.
4.4
Space Constraint for Improved Depth Completion
The depth profile of real-world scene is characterized by strong correlations local spatial level. Thus, these spatial correlations play important role in predicting depth values in the setting of depth completion from sparse lidar measurements.
The role the Shapley values to create a ranking of lines without any additional constraints. As we will present in the experiments in Sec. 5, the lines identified as most important via the Shapley values may be in close spatial proximity, which leaves other regions of the scene without any depth information, even though such information would greatly facilitate depth completion in those regions. Therefore, the spatial structure of the set of lidar lines plays key role in selecting lines, which motivates us adapt approach follows in order to take it into account.
5

---

Figure 1: Visualization of regions descrihed Sec. 4.4. We denote as the entire depth map and R as the minimal contiguous region that contains all the N lidar lines and S spread lines.
R {
D
Let the spatial extent of the depth map D be divided into D N sub-regions which describe the lidar lines such that |D| D. Let N E N be a fixed budget of lines. Then the lines which are neglected are K = D - N. Importantly, we note that the N lines can be "spread out" or huddled together. For example, when the N lines are all consecutive, they take up the space of one joined region of size N (with the remaining space also being joined in one or two regions). However, this is exactly the case which we wish to avoid, as the depth measurements are not spread out over the entire scene.
Consider the minimal contiguous region of the image which contains all selected lidar lines and denote it by R. This area R. may contain regions corresponding to lidar lines that are not selected, which we describe as spread, S. That is, S C R. The entire depth map D = R U R'. When all the lines are huddled together, RR| N, [R'| = and [S] O. When there is one line that is not selected between the selected lines, then [R] = N + with S = 1, and [R|| K
In the experiments of Sec. 5, we show empirically how increasing the spread S improves the quality of the depth map, even when the number of lines remains constant.
Experiments
For the experimental validation of our method, we use the KITTI benchmark. KITTI consists of driving scenes recorded from the car. The data consist of RGB images, sparse range measurements produced by lidar, and dense depth maps which are annotated as ground truth-depth images. To measure the performance of the depth completion task, we use the RMSE between the ground-truth depth and the predicted depth. Given sparse set of depth points, we perform depth completion using the network proposed in [5].
We first train the network from scratch for 10 epochs. Subsequently, we select subset of the lines according to given method. We distinguish several ways to select lines. "Shapley global" is static scheme, where the input in every frame consists of the same set of lines selected by the Shapley value. Shapley global is computed based on samples from the entire dataset. On the other hand, "Shapley local" performs the selection for every image separately (as given in Sec. 3). Here, to compute the SVs, multiple samples (350 in this case) are drawn for every image.
The second component of our method is the spati sampling strategy. The basic selection baseline selects equally spaced lines, starting from the top line (which is most significant according to our tests), c.g., line 64, 48, 32 and 16 for the case of 4, where is the number of selected lines. Subsequently, we combine the two components in two distinct variants. The first one, "SaS constant-k" fixes the least amount of space between the lines, that is any two selected lines need to be separated by at least l intermediate lines. In our experiments, we set 1. Then SaS constant-1 describes a set of lines selected with the Shapley ranking with the spatial constraint that two lines cannot be adjacent each other. The second variant, which we name "SaS flexible-s" proves to be the most effective. Here, spread budget of size is flexibly assigned between the signal lines. The details are given in the following.
Fig. 2 summarizes the performance of the variants described above for a range of line budgets. The number of the lines at our disposal influences the trade-off between the Shapley value of the selected lines and the degree of spatial coverage associated with them and determines large part the method and the potential performance. As Fig. shows, when the number of selected lines is larger than 50% of the total number of lines, the preserved lidar signal is dense and spatial constraints are not very important, since the sheer number of lines suffices to obtain an accurate result. This is also

---

Stapiey (ocal)
Method / # of lines
32
16
Shapley (local)
946
3215
5656
6620
Shapley (global)
1078
3872
5471
6848
9992
Spaced
1149
1951
4289
7671
Random
1799
2416
5043
10490
SaS constant-1 (local)
1361
1419
3183
5838
3003
SaS flexible (global)
888
1178
2147
3436
2000
20
Number lines
Figure 2: Performance comparison of lidar line selection methods for depth completion on KITTL The table on the left shows the RMSE (in mm) for the compared methods with varying number of selected lines. The plot on the right delves deeper into the comparison of two of the methods, Shapley (local) and SaS constant-1 (local), comparing them for all possible numbers of lines. The RMSE with 64 lines 853
140
0:00
130
50%
0.09
1099
2000
1209
25
(a) lidar lines
(b) 16 lidar lines
(c) 32 lidar lines
Figure 3: The effect of including space between the selected lines the quality of depth completion in terms of RMSE between the ground-truth and the predicted depth. For fixed number of lines N E [8, 16, 32]. the total spatial spread of them varied (horizontal axis). Spread is described via S, the number of lines which are located between the two extreme selected lines and are not selected. For varying S E [0, 64 - N S random lines are selected and removed from the range of lines [64 - N S, 64] (the average over 15 samples). Note that the topmost line is numbered as 64.
confirmed by Fig. 3, which studies SaS flexible and the role of space in depth. the case of 32 lines, average, the RMSE rises as the spread increases. In the table in Fig. we include the best sample which includes the amount of spread equal to 4 (the details of the exact spread configurations given in the Appendix). In general, though, when the number of available lines is > 32, the best option is to select the lines directly produced by the Shapley value. Let us note that with 32 lines both SaS and the local Shapley value reach similar performance to the original RMSE with 64 lines which is 853 mm.
As the number of selected lines decreases, the relative importance of incorporating spatial awareness in method increases. This importance is evident from the fact that simply selecting equally spaced lines yields better results than the Shapley value. The Shapley value itself selects lines which do not have adequate spatial coverage of the image and therefore do not produce a completed depth map of good quality. On the contrary, for small numbers of lines, incorporating a spatial spread constraint into the Shapley value scheme produces the best results. For and 8 lines, SaS substantially improves the performance of both of the plain Shapley value variants and the equally spaced baseline, however should note that when the number of available lines drops toward one-digit number, it is hard to expect to match the result of the 64 lines. On the other hand, the results show, even just quarter of the lines produces result which is acceptable. The visual results are presented in Fig. 4.
5.1 The Effect of the Spatial Constraint
As argued in Sec. 4.4, the space spread S between the input features plays role in depth completion. In this experiment we verify the role of space for the varying amount of spatial spread
7

---

RGB
GT
Sh-16 SaS-16 Sh-32
Figure 4: Examples of depth maps generated with the proposed methods with subset of lidar lines. Shapley local (Sh-32 and Sh- 16) uses 32 and 16 lines, respectively without the space constraint. SaS- 16 uses both the Shapley value and the space constraint. Notice the role of space when using 16 lines, that is smoother prediction over the entire depth map when utilising space constraint, car/bus in the bottom-lee corner. When using 32 lines, denser input yiclds more accurate predictions both for closer and further regions without the need to incorporate spatial constraints. With fewer lines, Shaplcy focuses more the far-away regions.
notice more accurate prediction the nearby
which is a simplified version of the "SaS flexible" method. Given a budget of N lines, we verify the impact of the spatial extent of the region as described Sec. 4.4 by varying the size of R. In the experiment, we fix N to 4, 8, 16 32, and we vary [R] E [N, 42) (42 the number visible lines). Since there are (() possible configurations, we randomly select indices for the lidar lines and verify the accuracy of the depth map inferred with the respective configuration of selected lines. The random sampling is repeated 15 times. The results are shown in Fig. 3.
Fig. 3 presents consistent results. As the number of available lidar lines increases, the role of spatial spread decreases. In the case of 32 lines, on average, including space actually deteriorates the result. Also in the case of and 16 lines, empirically, the improvement from spreading the lines noticeable up certain tipping point. Afterwards, the number of empty lines increases, the quality of depth map decreases, as the signal becomes too sparse compared to the space between the lines.
SaS flexible differs from SaS constant that we sample the top lines as given by the Shapley value ranking with higher probability assigned to lines which are ranked higher. As a result, for different samples, the amount spread varies. Thanks to the empirical observations from this section, given fixed number of lines, we allow for "spread budget" and test only the samples which do not exceed the budget. This allows to sieve through the line configurations, making less samples necessary to obtain the same level of performance, as presented in Fig. 2.
6 Conclusion
Motivated by lowering the costs of access lidar for widespread applications, we introduce the problem of lidar line selection for depth completion. Thanks to the algorithmic blend of the game- theoretical concept of the Shapley value and the spatial constraint necessitated by spatial scene correlations, our depth completion method predicts depth maps comparable quality with only a fraction of the lines. These global line configurations can be used to build a custom non-uniform sensor. In the future, aim further improve the results for instantaneous local selection to reduce the computational overhead required every frame. Morever, as we present the benefit of optimizing the selected lines, expect the design of future lidar sensors will start being influenced accordingly.
8

---

References
[1] L. S. Shapley. A value for n-person games. Contributions to the Theory of Games (AM-28), 1 (4):169, 1953.
[2] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
[3] Ma and S. Karaman. Sparse-to-dense: Depth prediction from sparse depth samples and single image. In IEEE international conference robotics and automation (ICRA), pages 4796-4803, 2018.
[4] M. Jaritz, R. de Charette, E. Wirbel, X. Perrotton, and F. Nashashibi. Sparse and dense data with CNNs: Depth completion and semantic segmentation. In International Conference on 3D Vision (3DV), 2018.
(5] Ma, G. Cavalheiro, and Karaman. Self-supervised sparse-to-dense: Self-supervised depth completion from lidar and monocular camera. In International Conference on Robotics and Automation (ICRA), pages
2019.
[6] J. Qiu, Z. Cui, Y. Zhang, X. Zhang, S. Liu, B. Zeng, and M. Pollefeys. DeepLiDAR: Deep surface normal guided depth prediction for outdoor scene from sparse LiDAR data and single color image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[7] Y. Xu, X. Zhu, Shi, G. Zhang, H. Bao, and H. Li. Depth completion from sparse lidar data with depth-normal constraints. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2019.
[8] W. Van Gansbeke, D. Neven, B. De Brabandere, and L. Van Gool. Sparse and noisy LiDAR completion with RGB guidance and uncertainty. In International Conference on Machine Vision Applications (MVA), 2019.
[9] Y. Yang. A. Wong, and S. Soatto. Dense depth posterior (DDP) from single image and sparse range. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[10] J. Park, K. Joo, Z. Hu, C.-K. Liu, and I. So Kweon. Non-local spatial propagation network for depth completion. In The European Conference Computer Vision (ECCV), 2020.
[11] X. Xiong, H. Xiong, K. Xian, C. Zhao, Z. Cao, and X. Li. Sparse-to-dense depth completion revisited: Sampling strategy and graph construction. In The European Conference on Computer Vision (ECCV), 2020.
[12] Zhao, M. Gong, H. Fu, and D. Tao. Adaptive context-aware multi-modas modal network for depth completion. IEEE Transactions Image Processing, 30:5264-5270 2021.
[13] J. Tang, F.-P. Tian, W. Feng. J. Li, and P. Tan. Learning guided convolutional network for depth completion. IEEE Transactions on Image Processing. 30:1 1011161111 1-166111111,,111 2021.
[14] M. Hu, S. Wang, B. Li. S. Ning, L. Fan, and X. Gong. PENet: Towards precise and efficient image guided depth completion. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
[15] K. Rho, J. Ha, and Y. Kim. GuideFormer: Transformers for image guided depth completion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.

---

[16] A. Wong and S. Soatto. Unsupervised depth completion with calibrated backprojection layers. In Proceedings of the IEEEJCVF International Conference on Computer Vision (ICCV), 2021.
[17] Linardatos, V. Papastefanopoulos, and S. Kotsiantis. Explainable AI: A review of machine learning interpretability methods. Entropy, 23(1), 2020.
[18] M. T. Ribeiro, S. Singh, and C. Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference knowledge discovery and data mining, pages 1135-1 144. ACM, 2016.
[19] S. M. Lundberg and S.-I. Lee. A unified approach to interpreting model predictions. In Advances Neural Information Processing Systems, 2017.
[20] M. Ancona, E. Ceolini, A. C. Öztireli, and M. Gross. A unified view of gradient-based attribution methods for deep neural networks. In NIPS Workshop on Interpreting. Explaining and Visualizing Deep Learning, 2017.
[21] J. Chen, L. Song, M. Wainwright, and M. Jordan. Learning to explain: An information-thhoreticc perspective model interpretation. In International Conference Machine Learning, 2018.
[22] Yoon, J. Jordon, and M. van der Schaar. INVASE: Instance-wise variable selection using neural networks. In International Conference on Learning Representations, 2018.
[23] A. Charnes, B. Golany, M. Keane, and J. Rousseau. Extremal principle solutions of games in characteristic function form: Core, chebychev and shapley value generalizations. 1988.
10


Reviews:
[{"review_id": "gwqh2E3t-Qu", "reviewer": "Reviewer_xfBr", "summary": "The authors introduce the problem of selecting a subset of lines from a lidar scan that maximizes the accuracy of the depth map one obtains when completing the lidar line selection to compare against the original complete depth map. A solution to this line subset finding problem is presented which selects scan lines one by one based on their marginal contribution to the current line selection while encouraging spatial spread of the selected scan lines. The proposed method is evaluated on the KITTI dataset in conjunction with a deep-learning-based depth completion network. Various adaptations of the approach, such as various spatial sampling techniques and whether the line selection occurs once for all frames (global) or at every frame (local), are evaluated on the RMSE metric against the ground-truth lidar scans.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe approach is technically sound and explained clearly by introducing the Shapley value and the proposed spatial sampling techniques.\n\nThe evaluation of the method on the KITTI benchmark is detailed, with extensive ablations over the spatial sampling strategy and performance results, albeit only the RMSE is considered as accuracy metric.\n\n# Weaknesses\n\nThe motivation for the introduced problem seems questionable to me. It is mentioned that \"lowering the costs of access to lidar\" (l. 280) motivated the proposed method. However, I do not see how it is economical to design new custom lidar hardware based on a non-uniform line pattern that has been found by this method when regular lidars measure equidistant scan lines. Do the authors have any evidence for the feasibility of lowering lidar cost through their approach?\nFuthermore, when the line selection is allowed to change at every frame (Shapley local) or even between datasets -- what is the practical benefit/application of this? If the computing time can be reduced through less input data to perception pipelines, is the proposed method fast enough to warrant this additional lidar scan \"downsampling\" step at every frame? When we focus our attention to downsampling, how does Shapley-local compare to other approaches in this area, such as voxel-based downsampling?\n\nEvaluating the method on a single dataset which has the same lidar placement accross all measurements potentially limits the generality of the proposed method. It would be interesting how the selection of lidar lines is affected by the tilt or height (for example) of the lidar, even on simulated data. Tying it back to the aforementioned practicality problem of designing custom lidar hardware -- is there any reason to assume that the same line selection will work globally under different conditions?\n\nThe paper does not discuss the choice of depth completion algorithm. Only in the experiment section is it mentioned that the approach from [5], a self-supervised deep learning method, is used as this crucial component which directly affects the performance of the overall pipeline. It should be explicitly said and better motivated in Sec. 2 that [5] is used as depth completion module. Furthermore, it has to be explained whether the network needs to be retrained for different line selections and what hyperparameters have been used. Do these hyperparameters influence which of the proposed line selection strategies yield the best results? If so, they must be included in the performance evaluations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I cannot recommend acceptance of this work in its current form because it is difficult to assess the relevance to robotics of the introduced problem space and the proposed method. The authors should find a more convincing angle to motivate line selection (both global and local) and consider positioning the paper as an approach of downsampling of lidar scans, which is well motivated and has many well established baselines to compare against.\n\nFuther issues to address to improve clarity and overall strength of the paper are given below.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "# Major Issues\n\nIt seems that the appendix is missing for this submission. In line 164 a \"supplement\" is mentioned to describe the criteria of distribution fairness. In line 245 the authors again refer to an appendix.\n\nPerformance numbers should be provided for the given experiments. In particular for the local approach where the line selection happens at every frame, it is important to know whether the approach can run in real time. On that note, how many binary vectors were sampled to evaluate the Shapley values for the line selection candidates?\n\nNo limitations of the proposed method have been discussed in the paper.\n\n# Minor Issues\n\nIt would be clearer to see how the method compares in Fig. 2 to only show the line diagram in Fig. 2 (right) without the table, and plot the evolution of the RMSE of all the methods.\n\nEq. 1 seems that it should not have the $\\arg$ operator in it."}}, {"review_id": "DIfEKXRcitc", "reviewer": "Reviewer_6id7", "summary": "This paper proposes a method for reducing the cost of accessing LiDAR for depth completion task. To achieve the saving, LiDAR lines are ranked with Shapley value. Lines with high Shapley values are considered important and kept while others are discarded.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- Method description is clear and easy to follow.\n- Modeling the importance of the LiDAR line with Shapley value looks natural and also promising.\n- Experiments for accuracy evaluation show the power of this method\n\nWeakness:\n- Abstract is hard to read\n- The objective of this method is ambiguous. What cost is saved with your method?\n  - Energy consumption for LiDAR? In my understanding, to assess the importance of the LiDAR lines, this method requires the data from all LiDAR lines. So no LiDAR lines can be completely shut down.\n  - Computation for depth completion? As far as I know, most depth completion methods project the point cloud collected from LiDAR to a 2D plane. So no matter how many LiDAR points there are, the inputs to the network are always 2D planes with the same number of pixels. The cost for inference doesn't change.\n- To assess the Shapley value, multiple samplings, inferences, and accuracy evaluations are required. The overhead of this sampling process seems to overwhelm the benefit of line selection, especially in the dynamic problem setting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The modeling part and the result of this paper look good. Though not supported by current LiDAR, the method proposed in this paper does show the probability of cost saving. Though the objective and overhead are ambiguous in the main text, they are clarified during the rebuttal period. Thus I change my score and recommend accepting this paper.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations section not present", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "The objective and the overhead mentioned in the weakness section.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "CYoJbgCUq5z", "reviewer": "Reviewer_3dHW", "summary": "This paper proposed a solution to select the most effective LiDAR lines for the depth completion task. Major contributions are as follows:\n\n(1) The author proposed to leverage the Shapley value for LiDAR lines to evaluate the marginal contribution to the overall depth completion output. \n(2) A spatially-aware Shapley value (SaS) scheme is proposed to take into account the spatial configuration of the line selection.\n(3) The proposed method has two basic variants of the approach, the global version which select lines at a dataset level and the local version which selects lines for each image independently.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* This paper proposed a novel method to approximate the marginal contribution of a LiDAR line to the depth completion task, which could provide valuable information for the sparse LiDAR design.\n**Weaknesses:**\n* In my understanding, the proposed method needs pre-training on the dense (64-line) LiDAR data before the line selection, which hinders its application on Sparse-LiDAR-only systems.\n* The computation cost analysis is missing.\n* It would be better if the line selection could be efficiently predicted only with the RGB image input, to guide the LiDAR design or selection.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The LiDAR line selection could greatly affect the depth completion performance because different lines contribute a different amount of importance. The problem author tried to solve is significant. Although the method and presentation are not perfect, it is still inspiring.\n\nPost-rebuttal: the proposed method seems not practical for current rotation-based LiDARs, but it is still inspiring and may be possible to guide the design of future solid-state LiDARs which could generate variational scanning patterns. I would keep my rating of \"weakly accept\"", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations section not present", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "**Major Issues:**\n* The limitation section is missing.\n* Analysis of computation cost is needed.\n**Minor Issues:**\n* The author could detailly state how the proposed method could contribute to the real-world application. If the line selection requires pre-training or sampling on dense LiDAR data, the value of the proposed method will need further justification.\n* There is more SOTA depth completion methods that could be discussed in the Related Works section like:\n\n[1] X. Cheng, P. Wang, C. Guan, and R. Yang. CSPN++: learning context and resource aware convolutional\nspatial propagation networks for depth completion. AAAI Press.\n\n[2] Choi, Jaehoon, et al. \"Selfdeco: Self-supervised monocular depth completion in challenging indoor environments.\" 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.\n\n[3] Feng, Ziyue, et al. \"Advancing self-supervised monocular depth learning with sparse liDAR.\" Conference on Robot Learning. PMLR, 2022.\n\n[4] Yoon, Sungho, and Ayoung Kim. \"Balanced depth completion between dense depth inference and sparse range measurements via kiss-gp.\" 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.\n\n\n"}}]

Author rebuttal:

Decision: Accept (poster)