Title: Is Your LiDAR Placement Optimized for 3D Scene Understanding?
Venue: NeurIPS 2024
Accepted: yes
Keywords: Autonomous Driving

LiDAR Semantic Segmentation

Sensor Placement
Abstract: The reliability of driving perception systems under unprecedented conditions is crucial for practical usage. Latest advancements have prompted increasing interest in multi-LiDAR perception. However, prevailing driving datasets predominantly utilize single-LiDAR systems and collect data devoid of adverse conditions, failing to capture the complexities of real-world environments accurately. Addressing these gaps, we proposed Place3D, a full-cycle pipeline that encompasses LiDAR placement optimization, data generation, and downstream evaluations. Our framework makes three appealing contributions. 1) To identify the most effective configurations for multi-LiDAR systems, we introduce the Surrogate Metric of the Semantic Occupancy Grids (M-SOG) to evaluate LiDAR placement quality. 2) Leveraging the M-SOG metric, we propose a novel optimization strategy to refine multi-LiDAR placements. 3) Centered around the theme of multi-condition multi-LiDAR perception, we collect a 280,000-frame dataset from both clean and adverse conditions. Extensive experiments demonstrate that LiDAR placements optimized using our approach outperform various baselines. We showcase exceptional results in both LiDAR semantic segmentation and 3D object detection tasks, under diverse weather and sensor failure conditions.

Is Your LiDAR Placement Optimized for 3D Scene Understanding?

Abstract

The reliability of driving perception systems under unprecedented conditions is crucial for practical usage. Latest advancements have prompted increasing interest in multi-LiDAR perception. However, prevailing driving datasets predominantly utilize single-LiDAR systems and collect data devoid of adverse conditions, failing to capture the complexities of real-world environments accurately. Addressing these gaps, we proposed Place3D , a full-cycle pipeline that encompasses LiDAR placement optimization, data generation, and downstream evaluations. Our framework makes three appealing contributions. 1) To identify the most effective configurations for multi-LiDAR systems, we introduce the Surrogate Metric of the Semantic Occupancy Grids (M-SOG) to evaluate LiDAR placement quality. 2) Leveraging the M-SOG metric, we propose a novel optimization strategy to refine multi-LiDAR placements. 3) Centered around the theme of multi-condition multi-LiDAR perception, we collect a 280,000-frame dataset from both clean and adverse conditions. Extensive experiments demonstrate that LiDAR placements optimized using our approach outperform various baselines. We showcase exceptional results in both LiDAR semantic segmentation and 3D object detection tasks, under diverse weather and sensor failure conditions.

1 Introduction

Accurate 3D perception plays a crucial role in autonomous driving, involving detecting the objects around the vehicle and segmenting the scene into meaningful semantic categories. LiDARs are becoming crucial for driving perception due to their capability to capture detailed geometric information about the surroundings [3, 109, 53, 54, 118]. While the latest models achieved promising accuracy on standard datasets, e.g. , nuScenes [7] and SemanticKITTI [6], improving the resilience of perception under corruptions and sensor failures is still a critical yet under-explored task [46, 94, 5, 95].

Recent studies have primarily focused on refining the sensing systems by designing new algorithms with novel model architectures [15] or 3D representations [28, 99, 11]. The selection of LiDAR configurations, however, often relies on industry experience and design aesthetics. Therefore, existing literature has potentially overlooked optimal LiDAR placements for maximum sensing efficacy.

An intuitive approach for optimizing LiDAR configurations involves a comprehensive cycle of data collection, model training, and validation across various LiDAR setups to enhance the autonomous driving system's perception accuracy [24, 52]. However, this approach faces significant challenges due to the substantial computational resources and extensive time required for data collection and processing [57, 17]. Although recent works [34, 65] made preliminary attempts to explore the impact

of LiDAR placements on perception accuracy, they have neither proposed an optimization method nor evaluated the performance in adverse conditions.

In this work, we delve into the optimization of sensor configurations for autonomous vehicles by tackling two critical sub-problems: 1) the performance evaluation of sensor-specific configurations, and 2) the optimization of these configurations for enhanced 3D perception tasks, encompassing 3D object detection and LiDAR semantic segmentation. To achieve this goal, we propose a systematic LiDAR placements evaluation and optimization framework, dubbed Place3D . The overall pipeline is endowed with the capability to synthesize point cloud data with customizable configurations and diverse conditions, including common corruptions, external disturbances, and sensor failures.

We first introduce an easy-to-compute Surrogate Metric of Semantic Occupancy Grids (M-SOG) to evaluate the equality of LiDAR placements. Next, we propose a novel optimization approach utilizing our surrogate metric based on Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to find the near-optimal LiDAR placements. To verify the correlation between our surrogate metric and assess the effectiveness of our optimization approach on both clean and adverse conditions, we design an automated multi-condition multi-LiDAR data simulation platform and establish a comprehensive benchmark consisting of a total of seven LiDAR placement baselines inspired by existing self-driving configuration from the autonomous vehicle companies.

Our benchmark, along with a large-scale multi-condition multi-LiDAR perception dataset, encompasses state-of-the-art learning-based perception models for 3D object detection [51, 107, 63, 109] and LiDAR semantic segmentation [18, 111, 86, 117, 50], as well as six distinct adverse conditions coped with weather and sensor failures [46, 95]. Utilizing the proposed framework, we explored how various perturbations and downstream 3D perception tasks affect optimization outcomes.

To summarize, this work makes the following key contributions:

To the best of our knowledge, Place3D serves as the first attempt at investigating the impact of multi-LiDAR placements for 3D semantic scene understanding in diverse conditions.

We introduce M-SOG, an innovative surrogate metric to effectively evaluate the quality of LiDAR placements for both detection (sparse 3D) and segmentation (dense 3D) tasks.

We propose a novel optimization approach utilizing our surrogate metric to refine LiDAR placements, which exhibit excellent LiDAR semantic segmentation and 3D object detection accuracy and robustness, outperforming baselines for relatively 9% in metrics.

We contribute a 280,000-frame multi-condition multi-LiDAR point cloud dataset and establish a comprehensive benchmark for LiDAR-based 3D scene understanding evaluations. We hope this work can lay a solid foundation for future research in this relevant field.

2 Related Work

LiDAR Sensing. LiDAR sensing is critical in autonomous vehicles, providing essential structural information for their operation [59, 3, 73]. Utilizing 3D data, LiDAR supports tasks such as semantic scene understanding [7, 26, 29, 6, 85, 48, 10], generation [75, 120, 74, 97], decision making [25, 8, 19, 87], and simulation [23, 68, 67, 93]. This work specifically targets LiDAR-based perception and simulation, which are at the forefront of current research in autonomous vehicle technology.

LiDAR-Based 3D Scene Understanding. LiDAR segmentation and 3D object detection are key tasks for 3D scene understanding. Segmentation models are categorized into point-based [88, 35, 36, 110, 77], range view [70, 90, 21, 98, 113, 16, 47, 2, 45, 99], bird's eye view [111, 115, 12], voxel-based [18, 86, 117, 33], and multi-view fusion [58, 38, 78, 60, 62, 14, 13, 61, 100] methods. While they show promising results on benchmarks, their performance across different LiDAR configurations is not well explored. For 3D object detection, models typically use point-based [82, 106, 105, 116] or voxel-based [114, 51, 69, 92, 83, 55, 66, 104] representations, with recent works favoring fusion for improved detection [64, 80, 81, 63, 56]. Our study complements these approaches by optimizing LiDAR placements to enhance performance under various real-world conditions.

LiDAR Perception Robustness. The reliability of LiDAR-based 3D scene understanding models in real-world scenarios is crucial [84, 39]. Recent studies have explored the robustness of 3D perception models against adversarial attacks [112, 96], common corruptions [46, 49, 94, 103, 43, 4], adverse weather [31, 30, 79, 37], sensor failures [108, 15, 28], and combined motion and sensor perturbations

L

W

H

Surrogate Metric of Semantic Occupancy Grids (M-SOG)

Laser Ray

= Covered Voxels

Map to P-SOG

(a)

3D Driving

Scenes

M-SOG

(d)

Evolution

Optimization

(b)

Probabilistic Semantic

Occupancy Grids

(c)

Compute Entropy

to Obtain M-SOG

Baseline

(e)

Multi-Condition

Multi-LiDAR

Dataset

3D Detection

& Segmentation

(f)

Perception

Performance

Evaluation

Optimizing LiDAR

Placements

C

CARLA

[102, 95]. To our knowledge, no prior work has focused on optimizing LiDAR placement to improve semantic 3D scene understanding robustness. Our Place3D addresses this gap by studying various placement strategies to enhance robustness in both in-domain and out-of-domain scenarios.

Sensor Placement Optimization. Optimizing sensor placements can enhance performance and address the challenges of heuristic design [42, 101]. In autonomous driving, optimizing LiDAR placements is relatively new [65]. Hu et al. [34] studied multi-LiDAR placements for 3D object detection, while Li et al. [57] examined LiDAR-camera configurations for multi-modal detection. Other works [41, 44, 9, 40] explored roadside LiDAR placements for V2X applications. Distinguishing from these efforts, our work is the first to investigate LiDAR placements for robust 3D scene understanding in challenging conditions. We establish benchmarks for both LiDAR semantic segmentation and 3D object detection, providing an in-depth analysis of optimal placements for robust perception.

3 Place3D: A Full-Cycle LiDAR Placement Pipeline

In this section, we introduce the Surrogate Metric of Semantic Occupancy Grids (M-SOG) to assess the 3D perception performance of sensor configurations (Section 3.2). Leveraging M-SOG, we propose a novel optimization strategy for multi-LiDAR placement (Section 3.3). Our approach is theoretically grounded; we provide an optimality certification to ensure that optimized placements are close to the global optimum (Section 3.4). Figure 1 depicts an overview of our Place3D framework.

3.1 Preliminary Formulation of Surrogate Metric

Region of Interest. Following the literature of sensor placement [34, 57], we define the Region of Interest (ROI) for perception as a finite 3D cuboid space [ l, w, h ] with ego-vehicle in the center, and divide the ROI space S into voxels with resolution [ δ l , δ w , δ h ] as S = { v 1 , v 2 , ..., v N } , where N = l δ l × w δ w × h δ h denotes the total number of voxels.

̸

Probabilistic Occupancy Grids (POG). Wedefine POG as the joint probability of all non-zero voxels in the ROI as p POG = p ( v 1 , v 2 , ..., v N ) = ∏ N i =1 , p ( v i ) =0 p ( v i ) , where each voxel is assumed to be independently and identically distributed. The voxel v i is occupied if it is within any 3D bounding box b c of object class c at frame t , which is denoted as v i = b ( t ) c . Therefore, the probability of being occupied for voxel v i among all T frames can be estimated as p ( v i ) = ∑ T t =1 1 ( v i = b ( t ) c ) /T .

Probabilistic Surrogate Metric. Now given some LiDAR placement L , the conditional POG can be found as the conditional joint distribution p POG | L = p ( v 1 , v 2 , ..., v N | L ) in the literature [34].

To show how much uncertainty the placement L can reduce in the ROI, the naive surrogate metric S-MIG [34] is introduced as the reduction of the entropy as -H POG | L = -∑ N i =1 H ( v i | L ) with constant H POG = ∑ N i =1 H ( v i ) , where H ( v i ) , H ( v i | L ) can be found through the entropy of Bernoulli distribution p ( v i ) , p ( v i | L ) , respectively.

3.2 M-SOG: Surrogate Metric of Semantic Occupancy Grids

The naive surrogate metric S-MIG [34] employs 3D bounding boxes as priors to understand 3D scene distribution. However, this approach exhibits substantial deviations from the actual physical boundaries of the objects and overlooks the occlusion relationships between objects and the environment in LiDAR applications. Furthermore, it is limited to detection tasks only. To overcome these limitations, we propose the Surrogate Metric of Semantic Occupancy Grids to assess and optimize LiDAR placement for 3D scene understanding tasks. Leveraging the Semantic Occupancy Grids derived from diverse environments, our approach can be expanded to tackling adverse conditions.

Semantic Occupancy Grids (SOG). Given a 3D driving scene with a set of semantic classes Y = { y 1 , y 2 , ..., y M } , where M represents the total number of semantic classes in the scene and assuming that each voxel can only be occupied by one semantic tag for each frame, we denote y ( t ) ( v i ) as the semantic class of voxel v i ∈ { v 1 , v 2 , ..., v N } at time frame t ∈ { 1 , 2 , ..., T } . Notably, empty voxels are also considered a semantic class. Accordingly, the set of voxels occupied by semantic class y c at frame t is defined as S ( t ) y c = { v i | y ( t ) ( v i ) = y c } , c = 1 , 2 , ..., M, t = 1 , 2 , ..., T . Then, we introduce SOG to describe the total semantic voxel distribution:

Probabilistic Semantic Occupancy Grids. In contrast to the Bernoulli distribution POG [34], we propose a more accurate Multinomial distribution: Probabilistic Semantic Occupancy Grids ( P-SOG ), denoted as p SOG , to represent the probability distribution of voxels belonging to certain semantic classes. Before estimating p SOG , we first traverse all frames from given scenes to obtain the probability ˆ p for each voxel v i occupied by each semantic class y c :

Notably, we denote voxel v i being occupied by semantic class y c as v i = y c , and ˆ p indicates estimated distributions from observed samples, whereas p refers to the statistical parameters to be estimated, which are unknown and non-random. We compute the joint probability of all non-zero voxels in the ROI to estimate the p SOG . Following the literature [34], the joint distribution over the voxel set S is:

̸

The uncertainty of Probabilistic Semantic Occupancy Grids reflects the 3D scene understanding capability of sensors within a given scene. To quantify this uncertainty, we calculate the entropy for the probability distribution of each voxel v i in ˆ p SOG as: ˆ H ( v i ) = -∑ M c =1 ˆ p ( v i = y c ) log ˆ p ( v i = y c ) .

Based on the independent and identically distributed assumption of SOG , the entropy of P-SOG over the voxel set S is given by: ˆ H SOG = E v i ∼ p S ∑ N i =1 ˆ H ( v i ) , where p S denotes the p SOG over the voxel set S . From the perspective of density estimation, the true P-SOG and its corresponding entropy can be estimated as p SOG = ˆ p SOG and H SOG = ˆ H SOG , respectively.

Surrogate Metric of Semantic Occupancy Grids. To evaluate LiDAR placements, we further analyze the joint probability distribution of voxels covered by LiDAR rays with varied LiDAR placements. Leveraging the Amanatides and Woo's Fast Voxel Traversal Algorithm [1], we obtain the voxels covered by LiDAR rays as S | L j = FastVT ( S, L j ) = { v L j 1 , v L j 2 , ..., v L j N j } given LiDAR configuration L = L j , j = 1 , 2 , ..., J , where j indexes the LiDAR placements, J is the number of total configurations, and N j is the number of voxels covered by rays of LiDAR configuration L = L j . Then, the semantic entropy distribution of P-SOG over the voxel set S | L j can be estimated as:

(a) Center

(b) Line

(c) Pyramid

(d) Square

(e) Trapezoid

(f) Line-roll

(g) Pyramid-roll

(h)

Ours

(i)

Ours

(Seg)

(j)

Ours

(2D)

We further define the information gain of 3D scene understanding as ∆ H = H SOG -H L j SOG to describe the perception capability of LiDAR configuration L j . Since H SOG is a constant given certain 3D semantic scenes with the fixed P-SOG distribution, we propose the normalized Surrogate Metric of Semantic Occupancy Grids (M-SOG) as follows to evaluate the perception capability:

3.3 Sensor Configuration Optimization

We adopt the heuristic optimization based on the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [32] to find an optimized LiDAR configuration.

Objective Function. We define the objective as F ( u j ) = M SOG ( L j ) , where u j ∈ U ⊂ R n represents the LiDAR configuration L j and is subject to the physical constraint P ( u j ) = 0 , and P ( u ) > 0 if it is violated, e.g. , mutual distance between LiDARs and distance from a 2D plane. In our optimization, u j includes the 3D coordinates and rolling angles of LiDARs, U is the finite cuboid space above the vehicle roof. Without ambiguity, we omit subscript j in the following text. We then transform the constrained optimization into the unconstrained Lagrangian form as G ( u ) = -F ( u )+ λP ( u ) , where λ is the Lagrange multiplier. We optimize G ( u ) through an iterative process that adapts the distribution of candidate solutions as Algorithm 1.

Optimization Approach. We first define a multivariate normal distribution N ( m ( k ) , ( σ ( k ) ) 2 C ( k ) ) , where m ( k ) , σ ( k ) , and C ( k ) are the mean vector, step size, and covariance matrix of the distribution of iteration k , respectively. We discretize the configuration space U with density δ and sample N k

candidates u ( k ) i ∼ N ( m ( k ) , ( σ ( k ) ) 2 C ( k ) ) in each iteration k , where i indexes the candidates. We update mean vector m k +1 for the next iteration k +1 as the updated center of the search distribution for LiDAR configuration. The overall process can be depicted as follows:

where M k is the number of best solutions we adopt to generate m ( k +1) , and w i are the weights based on solution fitness. We obtain the evolution path p ( k +1) C that accumulates information about the direction of successful steps as follows:

where c C is the learning rate for the covariance matrix update. The covariance matrix C controls the shape and orientation of the search distribution for LiDAR configurations. It can be updated at each iteration k in the following format:

Similarly, we update p σ as the evolution path for step size adaptation. Then, the global step size σ can be found below for the scale of search to balance exploration and exploitation:

where c σ is the learning rate for updating the evolution path p σ . d σ is a normalization factor to calibrate the pace at which the global step size is adjusted.

3.4 Theoretical Analysis

Once the evolution optimization empirically converges, it holds that the optimized solution is the local optima of the δ -density Grids space of LiDAR configuration space. In this section, we provide a stronger optimality certification to theoretically ensure that the optimized placement is close to the global optimum under the assumption of bounded and smooth objective function. Due to space limits, the full proof has been attached to the Appendix Section E.

Theorem 1 (Optimality Certification) . Given the continuous objective function G : R n → R with Lipschitz constant k G w.r.t. input u ∈ U ⊂ R n under ℓ 2 norm, suppose over a δ -density Grids subset S ⊂ U , the distance between the maximal and minimal of function G over S is upper-bounded by C M , and the local optima is u ∗ S = arg min u ∈ S G ( x ) , the following optimality certification regarding x ∈ U holds that:

where u ∗ is the global optima over U .

The global optimality certification Theorem 1 is applicable in practice because the Lipschitz constant k G and the distance between the maximal and minimal of objective function G over S can be approximated easily through calculating G ( u ( k ) i ) of Algorithm 1 for each sampled u ( k ) i over the δ -density Grids subset. Besides, we have a more general corollary below to further relax the assumption of bounded objective function.

Corollary 1. When U is a hyper-rectangle with the bounded ℓ 2 norm of domain U i ∈ R at each dimension i , with i = 1 , 2 , . . . , n , Theorem 1 can hold in a more general way by only assuming that the Lipschitz constant k G of the objective function is given, where Equation (11) becomes:

Compute

Probability

(a)

Raw Point Cloud

One Subset of All Frames

(b)

Dense

Merge

(c)

Semantic

Voxelize

(d)

Probabilistic SOG (P-SOG)

L

W

H

4 Experiments

4.1 Benchmark Setups

Data Generation. We generate LiDAR point clouds and ground truth using CARLA [23]. We use the maps of Towns 1, 3, 4, and 6 and set 6 ego-vehicle routes for each map. We incorporate 23 semantic classes for LiDAR semantic segmentation and 3 instance classes for 3D object detection. Data collection is performed for 10 LiDAR placements, resulting in a total of 280,000 frames: 1) For each placement, we gather 340 clean scenes and 360 corrupted scenes, with each scene consisting of 40 frames. 2) The clean set comprises 13,600 frames, including 11,200 samples (280 scenes) for training and 2,400 samples (60 scenes) for validation, following the split ratio used in nuScenes [7]. 3) The corruption set is used for robustness evaluation and contains 6 different conditions, each with 2,400 samples (60 scenes). More details are in the Appendix Section A.

LiDAR Configurations. We adopt five commonly employed heuristic LiDAR placements, which have been adopted by leading autonomous driving companies, as our baseline. These placements are represented in Figure 2a-e. Following KITTI [29], we configured the LiDAR sensor with a vertical field of view of [-24.8, 2.0] degrees. To achieve the Line-roll and Pyramid-roll configurations, we adjusted LiDAR roll angles for the Line and Pyramid setups, as depicted in Figure 2. We present detailed LiDAR configurations in the Appendix Section B.1.

Corruption Types. To replicate adverse conditions, we synthesized six types of corrupted point clouds on the validation set of each sub-set, following the settings of the Robo3D benchmark [46]. These corruptions can be categorized into 1) severe weather conditions, including "snow", "fog", and "wet ground", 2) external disturbances, including "motion blur", and 3) internal sensor failures, including "crosstalk" and "incomplete echo". We include the features and implementation details of corruptions in the Appendix Section B.2.

P-SOG Synthesis. We follow the steps as depicted in Figure 3 to generate semantic occupancy grids for each LiDAR scene. We first collect point clouds and semantic labels using high-resolution LiDARs and sequentially divide all samples into multiple subsets. Through the transformation of world coordinates of the ego-vehicle, frames of point clouds of each subset are aggregated into one frame of dense point cloud. Then, we utilize the voting strategy to determine the semantic label for each voxel in ROI and generate the SOG. This process is executed across all subsets to produce P-SOG. Notably, the P-SOG is only generated on the scenes of the training set.

Surrogate Metric of SOG. We compute the scores of M-SOG separately for 3D detection and segmentation, as shown in Table 1. To evaluate M-SOG for segmentation, we utilize all semantic classes to generate semantic occupancy grids. For detection, we specifically focus on the car semantic type, while merging the remaining semantic classes into a single category for M-SOG analysis.

Detector and Segmentor. For the benchmark, we conduct experiments using four LiDAR semantic segmentation models, i.e. , MinkUNet [18], SPVCNN [86], PolarNet [111], and Cylinder3D [117], and four 3D object detection models, i.e. , PointPillars [51], CenterPoint [107], BEVFusion-L [63], and FSTR [109]. The detailed training configurations are included in the Appendix Section B.3.

4.2 Comparative Study

We conduct benchmark studies to evaluate the performance of varied LiDAR placements in both clean and adverse conditions. We extensively examine the correlation between the proposed surrogate metric, known as M-SOG, and the final performance results. Through our analysis, we are able to demonstrate the effectiveness and robustness of the entire Place3D framework.

Effectiveness of M-SOG Surrogate Metric in Place3D. In Table 1, we report the scores of the M-SOG surrogate metric for various LiDAR placements. In Tables 2 and 3, we benchmark the 3D object detection and LiDAR semantic segmentation performance of varied LiDAR placements with state-of-the-art algorithms. Figure 4 illustrates the correlation between the proposed M-SOG

PolarNet

67.2

66.8

67.7

67.7

69.3

70.9

71.0

76.7

64

68

72

76

80

-3.5

-3.0

-2.5

-2.0

-1.5

-1.0

M-SOG(10

-6

)

mIoU (%)

(a) PolarNet

58.5

59.0

59.7

62.7

60.7

62.2

65.7

66.5

56

60

64

68

72

-3.5

-3.0

-2.5

-2.0

-1.5

-1.0

M-SOG(10

-6

)

mIoU (%)

MinkUNet

(b) MinkUNet

60.6

61.0

59.3

67.6

63.4

67.9

67.1

68.3

57

61

65

69

73

-3.5

-3.0

-2.5

-2.0

-1.5

-1.0

M-SOG(10

-6

)

mIoU (%)

SPVCNN

(c) SPVCNN

69.8

68.5

68.9

68.4

69.9

69.3

72.7

73.0

64

68

72

76

-3.5

-3.0

-2.5

-2.0

-1.5

-1.0

M-SOG(10

-6

)

mIoU (%)

Cylinder3D

(d) Cylinder3D

49.3

49.2

50.2

50.8

50.7

51.0

52.5

53.0

47

49

51

53

55

-1.7

-1.6

-1.5

-1.4

-1.3

-1.2

M-SOG (10

-6

)

mAP (%)

(a) BEVFusion (M-SOG)

50.2

49.3

50.7

49.2

50.8

53.0

51.0

52.5

48

50

52

54

-6.5

-6.0

-5.5

-5.0

-4.5

mAP (%)

S-MIG (10

3

)

(b) BEVFusion (S-MIG)

43.4

43.8

43.5

44.6

46.1

46.1

46.5

46.8

41

43

45

47

49

-1.7

-1.6

-1.5

-1.4

-1.3

-1.2

M-SOG(10

-6

)

mAP (%)

(c) PointPillars (M-SOG)

43.5

43.4

46.1

43.8

44.6

46.8

46.1

46.5

42

44

46

48

-6.5

-6.0

-5.5

-5.0

-4.5

S-MIG (10

3

)

mAP (%)

(d) PointPillars (S-MIG)

surrogate metric and perception capacity. The results demonstrate a clear correlation, where the performance generally improves as the M-SOG increases. While there might be fluctuations in some placements with specific algorithms, the overall relationship follows a linear correlation, highlighting the effectiveness of our M-SOG for computationally efficient sensor optimization purposes. We show more plots in the Appendix Sections C.1 and C.2.

Comparisons to Existing Sensor Placement Methods. The effectiveness of LiDAR sensor placement metrics lies in the linear correlation between metrics and actual performance. We set the same ROI size and conduct a quantitative comparison with S-MIG [34] on 3D detection (as S-MIG [34] only works for 3D detection task). As shown in Figure 5, our metric exhibits better linear correlation. Since we introduce semantic occupancy information as a prior for the evaluation metric, our method more accurately characterizes the boundaries in the 3D semantic scene and addresses the issue of objects and environment occlusions when using 3D bounding boxes as priors. Moreover, our LiDAR placement method makes the first attempt to include both detection and segmentation tasks. We present additional comparisons in the Appendix Section C.3.

Superiority of Optimization via Place3D. As shown in Tables 2 and 3, our optimized configurations achieved the best performance in both segmentation and detection tasks under both clean and adverse conditions among all models. We report per-class quantitative results in the Appendix Sections C.1 and C.2. The improvement from optimization remains significant even when comparing our optimized configurations against the best-performing baseline in each task. For segmentation, optimized placements outperform the best-performing baseline by up to 0.8% on clean datasets and by as much as 1.5% on corruption. For detection, optimized configurations exceed the best-performing baseline by up to 0.7% on clean datasets and by up to 0.3% on corruption.

Robustness of Optimization via Place3D. The M-SOG utilizes semantic occupancy as prior knowledge, which is invariant to changing conditions, thereby enabling robustness of optimized placement in adverse weather. Under the corrupted setting, although the correlation between M-SOG and perception performance is not as obvious as that in the clean setting and shows some fluctuation, our optimized LiDAR configuration consistently maintained its performance in adverse conditions, mirroring its effectiveness in the clean condition. While the top-performing baseline LiDAR configurations in clean datasets might be notably worse compared to others when faced with corruption, the optimized configuration via Place3D consistently shows the best performance under adverse conditions. We showcase several qualitative results on this aspect in the Appendix Section D.

Intuitive Interpretation. We observe several intuitive reasons why refined sensor placement is beneficial in our experiments. 1 ) LiDAR heights. Increasing the average height of the 4 LiDARs improves performance, likely due to an expanded field of view. 2 ) Variation in heights. A greater

47.4

48.6

48.6

47.3

49.8

49.0

51.1

47.9

46

48

50

52

54

PolarNet

MinkUNet

SPVCNN Cylinder3D

Ours

Ours (Corrupt Optimized)

(a) Against Corrupt

66.8

59.0

61.0

68.5

67.7

59.7

59.3

68.9

69.3

60.7

63.4

69.9

71.2

62.4

65.0

70.3

58

63

68

73

78

PolarNet

MinkUNet

SPVCNN

Cylinder3D

Trapezoid

Line

Square

Ours 2D

(b) 2D Placements (Clean)

39.3

44.5

43.8

42.0

40.6

47.6

41.5

44.8

38.2

48.4

46.6

43.4

44.5

48.5

51.5

46.5

35

41

47

53

59

PolarNet

MinkUNet

SPVCNN

Cylinder3D

Trapezoid

Line

Square

Ours 2D

(c) 2D Placements (Corrupt)

difference in LiDAR heights enhances perception, as varied height distributions capture richer features from the sides of objects. 3 ) Uniform distribution. The pyramid placement performs better in segmentation, as a more spread-out and uniform distribution of 4 LiDARs captures a detailed surface.

4.3 Ablation Study

In this section, we further analyze the interplay between our proposed optimization strategy and perception performance to address two questions: 1 ) How does our optimization strategy inform LiDAR placements to improve robustness against various forms of corruption? 2 ) How can our optimization strategy enhance perception performance in scenarios with limited placement options?

Optimizing Against Corruptions. In the previous subsection, we deploy the placement optimized on the clean point clouds directly on corrupted ones to assess the robustness. To further showcase the capability of our method against corruptions, we compute the M-SOG scores utilizing the PSOG derived from the corrupted semantic point clouds under adverse conditions and perform the optimization process, as shown in Figure 6a. Quantitatively compared with Ours in Table 2, the configurations derived from optimization on adverse data outperform the configurations generated solely on clean data. These results indicate that customizable optimization tailored for adverse settings is an effective approach to enable robust 3D perception.

Optimizing 2D Placements. Placing LiDARs at different heights on the roof might affect automotive aesthetics and aerodynamics. Therefore, we investigate the effect of our optimization algorithm in the presence of constraints, by limiting the LiDARs to the same height and considering only 2D placements. We fix the height of all LiDARs to find the optimal placement on the horizontal plane of the vehicle roof. We use the Line , Square , and Trapezoid placements in Figure 2 for comparison. Experimental results in Figures 6b and 6c show that our optimized LiDAR placements outperform baseline placements at the same LiDAR height on both clean and corruption settings.

Influence of LiDAR Roll Angles. Fine-tuning the orientation angles of the LiDARs on autonomous vehicles is an effective strategy to minimize blind spots and broaden the perception range. For 3D segmentation, Pyramid-roll slightly outperforms Pyramid , whereas Line-roll falls short of Line , which is highly aligned with scores of M-SOG in Table 1. For 3D detection tasks, the situation is reversed, but still consistent with the M-SOG results. This suggests that adjusting the LiDAR's angle can have varying impacts on the performance of 3D object detection and LiDAR semantic segmentation.

5 Conclusion

In this work, we presented Place3D , a comprehensive and systematic LiDAR placement evaluation and optimization framework. We introduced the Surrogate Metric of Semantic Occupancy Grids (M-SOG) as a novel metric to assess the impact of various LiDAR placements. Building on this metric, we culminated an optimization approach for LiDAR configuration that significantly enhances LiDAR semantic segmentation and 3D object detection performance. We validate the effectiveness of our optimization approach through a multi-condition multi-LiDAR point cloud dataset and establish a comprehensive benchmark for evaluating both baseline and our optimized LiDAR placements on detection and segmentation in diverse conditions. Our optimized placements demonstrate superior robustness and perception capabilities, outperforming all baseline configurations. By shedding light on refining the robustness of LiDAR placements for both tasks under diverse driving conditions, we anticipate that our work will pave the way for further advancements in this field of research.

Acknowledgments

This work was supported by the Robotics Department at the University of Michigan, Ann Arbor.

References

John Amanatides and Andrew Woo. A fast voxel traversal algorithm for ray tracing. Eurographics , 87(3):3-10, 1987.

Angelika Ando, Spyros Gidaris, Andrei Bursuc, Gilles Puy, Alexandre Boulch, and Renaud Marlet. Rangevit: Towards vision transformers for 3d semantic segmentation in autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5240-5250, 2023.

Claudine Badue, Rânik Guidolini, Raphael Vivacqua Carneiro, Pedro Azevedo, Vinicius B. Cardoso, Avelino Forechi, Luan Jesus, Rodrigo Berriel, Thiago M. Paixão, Filipe Mutz, Lucas de Paula Veronese, Thiago Oliveira-Santos, and Alberto F. De Souza. Self-driving cars: A survey. Expert Systems with Applications , 165:113816, 2021.

Till Beemelmanns, Quan Zhang, and Lutz Eckstein. Multicorrupt: A multi-modal robustness dataset and benchmark of lidar-camera fusion for 3d object detection. In IEEE Intelligent Vehicles Symposium , pages 3255-3261, 2024.

Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Jürgen Gall, and Cyrill Stachniss. Towards 3d lidar-based semantic scene understanding of 3d point cloud sequences: The semantickitti dataset. International Journal of Robotics Research , 40:959-96, 2021.

Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Juergen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In IEEE/CVF International Conference on Computer Vision , pages 9297-9307, 2019.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11621-11631, 2020.

Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106.11810 , 2021.

Xinyu Cai, Wentao Jiang, Runsheng Xu, Wenquan Zhao, Jiaqi Ma, Si Liu, and Yikang Li. Analyzing infrastructure lidar placement with realistic lidar simulation library. In International Conference on Robotics and Automation , pages 5581-5587, 2023.

Anh-Quan Cao and Raoul De Charette. Monoscene: Monocular 3d semantic scene completion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3991-4001, 2022.

Anh-Quan Cao, Angela Dai, and Raoul de Charette. Pasco: Urban 3d panoptic scene completion with uncertainty awareness. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14554-14564, 2024.

Qi Chen, Sourabh Vora, and Oscar Beijbom. Polarstream: Streaming lidar object detection and segmentation with polar pillars. In Advances in Neural Information Processing Systems , volume 34, pages 26871-26883, 2021.

Runnan Chen, Youquan Liu, Lingdong Kong, Nenglun Chen, Xinge Zhu, Yuexin Ma, Tongliang Liu, and Wenping Wang. Towards label-free scene understanding by vision foundation models. In Advances in Neural Information Processing Systems , volume 36, pages 75896-75910, 2023.

Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7020-7030, 2023.

Siran Chen, Yue Ma, Yu Qiao, and Yali Wang. M-bev: Masked bev perception for robust autonomous driving. In AAAI Conference on Artificial Intelligence , pages 1183-1191, 2023.

Huixian Cheng, Xianfeng Han, and Guoqiang Xiao. Cenet: Toward concise and efficient lidar semantic segmentation for autonomous driving. In IEEE International Conference on Multimedia and Expo , pages 1-6, 2022.

Huixian Cheng, Xianfeng Han, and Guoqiang Xiao. Transrvnet: Lidar semantic segmentation with transformer. IEEE Transactions on Intelligent Transportation Systems , 24(6):5895-5907, 2023.

Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3075-3084, 2019.

Laurene Claussmann, Marc Revilloud, Dominique Gruyer, and Sébastien Glaser. A review of motion planning for highway autonomous driving. IEEE Transactions on Intelligent Transportation Systems , 21(5):1826-1848, 2019.

MMDetection3D Contributors. MMDetection3D: OpenMMLab next-generation platform for general 3D object detection. https://github.com/open-mmlab/mmdetection3d , 2020.

Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy. Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds. In International Symposium on Visual Computing , pages 207-222, 2020.

Cruise. San francisco seeks stop sign on driverless cars. https://calmatters.org/newsletters/w hatmatters/2023/08/driverless-cars-california , Accessed: 2024-03-06, 2022.

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on Robot Learning , pages 1-16, 2017.

Bertrand Douillard, James Underwood, Noah Kuntz, Vsevolod Vlaskine, Alastair Quadros, Peter Morton, and Alon Frenkel. On the segmentation of 3d lidar point clouds. In IEEE International Conference on Robotics and Automation , pages 2798-2805, 2011.

Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R. Qi, Yin Zhou, Zoey Yang, Aurélien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, and Dragomir Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In IEEE/CVF International Conference on Computer Vision , pages 9710-9719, 2021.

Whye Kit Fong, Rohit Mohan, Juana Valeria Hurtado, Lubing Zhou, Holger Caesar, Oscar Beijbom, and Abhinav Valada. Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking. IEEE Robotics and Automation Letters , 7:3795-3802, 2022.

Ford. How ford's next-gen test vehicle lays the foundation for our self-driving business. https: //medium.com/self-driven/how-fords-next-gen-test-vehicle-lays-the-foundation-f or-our-self-driving-business-aadbf247b6ce , Accessed: 2024-03-06, 2020.

Chongjian Ge, Junsong Chen, Enze Xie, Zhongdao Wang, Lanqing Hong, Huchuan Lu, Zhenguo Li, and Ping Luo. Metabev: Solving sensor failures for 3d detection and map segmentation. In IEEE/CVF International Conference on Computer Vision , pages 8721-8731, 2023.

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3354-3361, 2012.

Martin Hahner, Christos Sakaridis, Mario Bijelic, Felix Heide, Fisher Yu, Dengxin Dai, and Luc Van Gool. Lidar snowfall simulation for robust 3d object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16364-16374, 2022.

Martin Hahner, Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Fog simulation on real lidar point clouds for 3d object detection in adverse weather. In IEEE/CVF International Conference on Computer Vision , pages 15283-15292, 2021.

Nikolaus Hansen. The cma evolution strategy: A tutorial. arXiv preprint arXiv:1604.00772 , 2016.

Fangzhou Hong, Lingdong Kong, Hui Zhou, Xinge Zhu, Hongsheng Li, and Ziwei Liu. Unified 3d and 4d panoptic segmentation via dynamic shifting networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 46(5):3480-3495, 2024.

Hanjiang Hu, Zuxin Liu, Sharad Chitlangia, Akhil Agnihotri, and Ding Zhao. Investigating the impact of multi-lidar placement on object detection for autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2550-2559, 2022.

Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11108-11117, 2020.

Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Learning semantic segmentation of large-scale point clouds with random sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(11):8338-8354, 2022.

Yihao Huang, Kaiyuan Yu, Qing Guo, Felix Juefei-Xu, Xiaojun Jia, Tianlin Li, Geguang Pu, and Yang Liu. Improving robustness of lidar-camera fusion model against weather corruption from fusion strategy perspective. arXiv preprint arXiv:2402.02738 , 2024.

Maximilian Jaritz, Tuan-Hung Vu, Raoul de Charette, Emilie Wirbel, and Patrick Pérez. xmuda: Crossmodal unsupervised domain adaptation for 3d semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12605-12614, 2020.

Peng Jiang, Philip Osteen, Maggie Wigness, and Srikanth Saripallig. Rellis-3d dataset: Data, benchmarks and analysis. In IEEE International Conference on Robotics and Automation , pages 1110-1116, 2021.

Wentao Jiang, Hao Xiang, Xinyu Cai, Runsheng Xu, Jiaqi Ma, Yikang Li, Gim Hee Lee, and Si Liu. Optimizing the placement of roadside lidars for autonomous driving. In IEEE/CVF International Conference on Computer Vision , pages 18381-18390, 2023.

Shaojie Jin, Ying Gao, Fei Hui, Xiangmo Zhao, Cheng Wei, Tao Ma, and Weihao Gan. A novel information theory-based metric for evaluating roadside lidar placement. IEEE Sensors Journal , 22(21):21009-21023, 2022.

Siddharth Joshi and Stephen Boyd. Sensor selection via convex optimization. IEEE Transactions on Signal Processing , 57(2):451-462, 2008.

O˘ guzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3d common corruptions and data augmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8963-18974, 2022.

Tae-Hyeong Kim, Gi-Hwan Jo, Hyeong-Seok Yun, Kyung-Su Yun, and Tae-Hyoung Park. Placement method of multiple lidars for roadside infrastructure in urban environments. Sensors , 23(21):8808, 2023.

Lingdong Kong, Youquan Liu, Runnan Chen, Yuexin Ma, Xinge Zhu, Yikang Li, Yuenan Hou, Yu Qiao, and Ziwei Liu. Rethinking range view representation for lidar segmentation. In IEEE/CVF International Conference on Computer Vision , pages 228-240, 2023.

Lingdong Kong, Youquan Liu, Xin Li, Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robo3d: Towards robust and reliable 3d perception against corruptions. In IEEE/CVF International Conference on Computer Vision , pages 19994-20006, 2023.

Lingdong Kong, Niamul Quader, and Venice Erin Liong. Conda: Unsupervised domain adaptation for lidar segmentation via regularized domain concatenation. In IEEE International Conference on Robotics and Automation , pages 9338-9345, 2023.

Lingdong Kong, Jiawei Ren, Liang Pan, and Ziwei Liu. Lasermix for semi-supervised lidar semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2170521715, 2023.

Lingdong Kong, Shaoyuan Xie, Hanjiang Hu, Lai Xing Ng, Benoit R. Cottereau, and Wei Tsang Ooi. Robodepth: Robust out-of-distribution depth estimation under corruptions. In Advances in Neural Information Processing Systems , volume 36, pages 21298-21342, 2023.

Lingdong Kong, Xiang Xu, Jiawei Ren, Wenwei Zhang, Liang Pan, Kai Chen, Wei Tsang Ooi, and Ziwei Liu. Multi-modal data-efficient 3d scene understanding for autonomous driving. arXiv preprint arXiv:2405.05258 , 2024.

Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12697-12705, 2019.

Li Li, Hubert P. H. Shum, and Toby P. Breckon. Less is more: Reducing task and model complexity for 3d point cloud semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9361-9371, 2023.

Rong Li, Raoul de Charette, and Anh-Quan Cao. Coarse3d: Class-prototypes for contrastive learning in weakly-supervised 3d point cloud segmentation. In British Machine Vision Conference , 2022.

Rong Li, Shijie Li, Xieyuanli Chen, Teli Ma, Wang Hao, Juergen Gall, and Junwei Liang. Tfnet: Exploiting temporal cues for fast and accurate lidar semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 4547-4556, 2024.

Xin Li, Tao Ma, Yuenan Hou, Botian Shi, Yucheng Yang, Youquan Liu, Xingjiao Wu, Qin Chen, Yikang Li, Yu Qiao, and Liang He. Logonet: Towards accurate 3d object detection with local-to-global cross-modal fusion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17524-17534, 2023.

Xin Li, Botian Shi, Yuenan Hou, Xingjiao Wu, Tianlong Ma, Yikang Li, and Liang He. Homogeneous multi-modal feature fusion and interaction for 3d object detection. In European Conference on Computer Vision , pages 691-707, 2022.

Ye Li, Hanjiang Hu, Zuxin Liu, Xiaohao Xu, Xiaonan Huang, and Ding Zhao. Influence of camera-lidar configuration on 3d object detection for autonomous driving. In IEEE International Conference on Robotics and Automation , pages 9018-9025, 2024.

Venice Erin Liong, Thi Ngoc Tho Nguyen, Sergi Widjaja, Dhananjai Sharma, and Zhuang Jie Chong. Amvnet: Assertion-based multi-view fusion network for lidar semantic segmentation. arXiv preprint arXiv:2012.04934 , 2020.

Mingyu Liu, Ekim Yurtsever, Xingcheng Zhou, Jonathan Fossaert, Yuning Cui, Bare Luka Zagar, and Alois C. Knoll. A survey on autonomous driving datasets: Data statistic, annotation, and outlook. arXiv preprint arXiv:2401.01454 , 2024.

Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment any point cloud sequences by distilling vision foundation models. In Advances in Neural Information Processing Systems , volume 36, pages 37193-37229, 2023.

Youquan Liu, Lingdong Kong, Xiaoyang Wu, Runnan Chen, Xin Li, Liang Pan, Ziwei Liu, and Yuexin Ma. Multi-space alignments towards universal lidar segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14648-14661, 2024.

Youquan Liu, Xin Li Runnan Chen, Lingdong Kong, Yuchen Yang, Zhaoyang Xia, Yeqi Bai, Xinge Zhu, Yuexin Ma, Yikang Li, Yu Qiao, and Yuenan Hou. Uniseg: A unified multi-modal lidar segmentation network and the openpcseg codebase. In IEEE/CVF International Conference on Computer Vision , pages 21662-21673, 2023.

Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L. Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. In IEEE International Conference on Robotics and Automation , pages 2774-2781, 2023.

Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel cnn for efficient 3d deep learning. In Advances in Neural Information Processing Systems , volume 32, pages 965-975, 2019.

Zuxin Liu, Mansur Arief, and Ding Zhao. Where should we place lidars on the autonomous vehicle?an optimal design approach. In IEEE International Conference on Robotics and Automation , pages 2793-2799, 2019.

Tao Ma, Xuemeng Yang, Hongbin Zhou, Xin Li, Botian Shi, Junjie Liu, Yuchen Yang, Zhizheng Liu, Liang He, Yu Qiao, et al. Detzero: Rethinking offboard 3d object detection with long-term sequential point clouds. In IEEE/CVF International Conference on Computer Vision , pages 6736-6747, 2023.

Sivabalan Manivasagam, Ioan Andrei Bârsan, Jingkang Wang, Ze Yang, and Raquel Urtasun. Towards zero domain gap: A comprehensive study of realistic lidar simulation for autonomy testing. In IEEE/CVF International Conference on Computer Vision , pages 8272-8282, 2023.

Sivabalan Manivasagam, Shenlong Wang, Kelvin Wong, Wenyuan Zeng, Mikita Sazanovich, Shuhan Tan, Bin Yang, Wei-Chiu Ma, and Raquel Urtasun. Lidarsim: Realistic lidar simulation by leveraging the real world. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11167-11176, 2020.

Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In IEEE/CVF International Conference on Computer Vision , pages 3164-3173, 2021.

Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss. Rangenet++: Fast and accurate lidar semantic segmentation. In IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 4213-4220, 2019.

Momenta. Several key members from oppo's zeku team joins momenta. https://pandaily.com/sev eral-key-members-from-oppos-zeku-team-joins-momenta , Accessed: 2024-03-06, 2023.

Motional. Uber launches hailing of motional robotaxis in las vegas. https://www.iottechnews.co m/news/2022/dec/07/uber-launches-hailing-motional-robotaxis-las-vegas , Accessed: 2024-03-06, 2022.

Khan Muhammad, Amin Ullah, Jaime Lloret, Javier Del Ser, and Victor Hugo C. de Albuquerque. Deep learning for safe autonomous driving: Current challenges and future directions. IEEE Transactions on Intelligent Transportation Systems , 22(7):4316-4336, 2020.

Kazuto Nakashima, Yumi Iwashita, and Ryo Kurazume. Generative range imaging for learning scene priors of 3d lidar data. In IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1256-1266, 2023.

Kazuto Nakashima and Ryo Kurazume. Learning to drop points for lidar scan synthesis. In IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 222-229, 2021.

Pony.ai. Pony.ai begins fully driverless testing in california. https://www.autonomousvehicleint ernational.com/news/testing/pony-ai-begins-fully-driverless-testing-in-califor nia.html , Accessed: 2024-03-06, 2022.

Gilles Puy, Alexandre Boulch, and Renaud Marlet. Using a waffle iron for automotive point cloud semantic segmentation. In IEEE/CVF International Conference on Computer Vision , pages 3379-3389, 2023.

Corentin Sautier, Gilles Puy, Spyros Gidaris, Alexandre Boulch, Andrei Bursuc, and Renaud Marlet. Image-to-lidar self-supervised distillation for autonomous driving data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9891-9901, 2022.

Alvari Seppänen, Risto Ojala, and Kari Tammi. 4denoisenet: Adverse weather denoising from adjacent point clouds. IEEE Robotics and Automation Letters , 8(1):456-463, 2022.

Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pvrcnn: Point-voxel feature set abstraction for 3d object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10529-10538, 2020.

Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn++: Point-voxel feature set abstraction with local vector representation for 3d object detection. International Journal of Computer Vision , 131(2):531-551, 2023.

Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrcnn: 3d object proposal generation and detection from point cloud. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 770-779, 2019.

Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(8):2647-2664, 2020.

Ziying Song, Lin Liu, Feiyang Jia, Yadan Luo, Guoxin Zhang, Lei Yang, Li Wang, and Caiyan Jia. Robustness-aware 3d object detection in autonomous driving: A review and outlook. arXiv preprint arXiv:2401.06542 , 2024.

Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2446-2454, 2020.

Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point-voxel convolution. In European Conference on Computer Vision , pages 685-702, 2020.

Siyu Teng, Xuemin Hu, Peng Deng, Bai Li, Yuchen Li, Yunfeng Ai, Dongsheng Yang, Lingxi Li, Zhe Xuanyuan, Fenghua Zhu, and Long Chen. Motion planning for autonomous driving: The state of the art and future perspectives. IEEE Transactions on Intelligent Vehicles , 8(6):3692-3711, 2023.

Hugues Thomas, Charles R. Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J. Guibas. Kpconv: Flexible and deformable convolution for point clouds. In IEEE/CVF International Conference on Computer Vision , pages 6411-6420, 2019.

Toyota. Ad/adas technology. https://woven.toyota/en/ad-adas-technology , Accessed: 2024-03-06, 2024.

Larissa T Triess, David Peter, Christoph B Rist, and J Marius Zöllner. Scan-based semantic segmentation of lidar point clouds: An experimental study. In IEEE Intelligent Vehicles Symposium , pages 1116-1121, 2020.

Waymo. Waymo to provide fully driverless rides in san francisco, california. https://www.theverge .com/2022/11/19/23467784/waymo-provide-fully-driverless-rides-san-francisco-c alifornia , Accessed: 2024-03-06, 2022.

Yi Wei, Zibu Wei, Yongming Rao, Jiaxin Li, Jie Zhou, and Jiwen Lu. Lidar distillation: bridging the beam-induced domain gap for 3d object detection. In European Conference on Computer Vision , pages 179-195, 2022.

Aoran Xiao, Jiaxing Huang, Dayan Guan, Fangneng Zhan, and Shijian Lu. Transfer learning from synthetic to real lidar point cloud for semantic segmentation. In AAAI Conference on Artificial Intelligence , pages 2795-2803, 2022.

Shaoyuan Xie, Lingdong Kong, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robobev: Towards robust bird's eye view perception under corruptions. arXiv preprint arXiv:2304.06719 , 2023.

Shaoyuan Xie, Lingdong Kong, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Benchmarking and improving bird's eye view perception robustness in autonomous driving. arXiv preprint arXiv:2405.17426 , 2024.

Shaoyuan Xie, Zichao Li, Zeyu Wang, and Cihang Xie. On the adversarial robustness of camera-based 3d object detection. Transactions on Machine Learning Research , 2024.

Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, and Raquel Urtasun. Learning compact representations for lidar completion and generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1074-1083, 2023.

Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. In European Conference on Computer Vision , pages 1-19, 2020.

Xiang Xu, Lingdong Kong, Hui Shuai, and Qingshan Liu. Frnet: Frustum-range networks for scalable lidar segmentation. arXiv preprint arXiv:2312.04484 , 2023.

Xiang Xu, Lingdong Kong, Hui Shuai, Wenwei Zhang, Liang Pan, Kai Chen, Ziwei Liu, and Qingshan Liu. 4d contrastive superflows are dense 3d representation learners. In European Conference on Computer Vision , pages 58-80, 2024.

Xiaohao Xu, Zihao Du, Huaxin Zhang, Ruichao Zhang, Zihan Hong, Qin Huang, and Bin Han. Optimization of forcemyography sensor placement for arm movement recognition. In IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 9845-9850, 2022.

Xiaohao Xu, Tianyi Zhang, Sibo Wang, Xiang Li, Yongqi Chen, Ye Li, Bhiksha Raj, Matthew JohnsonRoberson, and Xiaonan Huang. Customizable perturbation synthesis for robust slam benchmarking. arXiv preprint arXiv:2402.08125 , 2024.

Xu Yan, Chaoda Zheng, Ying Xue, Zhen Li, Shuguang Cui, and Dengxin Dai. Benchmarking the robustness of lidar semantic segmentation models. International Journal of Computer Vision , 132:2674-2697, 2024.

Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors , 18(10):3337, 2018.

Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11040-11048, 2020.

Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dense 3d object detector for point cloud. In IEEE/CVF International Conference on Computer Vision , pages 1951-1960, 2019.

Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11784-11793, 2021.

Kaicheng Yu, Tang Tao, Hongwei Xie, Zhiwei Lin, Zhongwei Wu, Zhongyu Xia, Tingting Liang, Haiyang Sun, Jiong Deng, Dayang Hao, Yongtao Wang, Xiaodan Liang, and Bing Wang. Benchmarking the robustness of lidar-camera fusion for 3d object detection. arXiv preprint arXiv:2205.14951 , 2022.

Diankun Zhang, Zhijie Zheng, Haoyu Niu, Xueqing Wang, and Xiaojun Liu. Fully sparse transformer 3d detector for lidar point cloud. IEEE Transactions on Geoscience and Remote Sensing , 61:1-12, 2023.

Tunhou Zhang, Mingyuan Ma, Feng Yan, Hai Li, and Yiran Chen. Pids: Joint point interaction-dimension search for 3d point cloud. In IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1298-1307, 2023.

Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, and Hassan Foroosh. Polarnet: An improved grid representation for online lidar point clouds semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9601-9610, 2020.

Yifan Zhang, Junhui Hou, and Yixuan Yuan. A comprehensive study of the robustness for lidar-based 3d object detectors against adversarial attacks. International Journal of Computer Vision , 132(5):1592-1624, 2024.

Yiming Zhao, Lin Bai, and Xinming Huang. Fidnet: Lidar point cloud semantic segmentation with fully interpolation decoding. In IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 4453-4458, 2021.

Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4490-4499, 2018.

Zixiang Zhou, Yang Zhang, and Hassan Foroosh. Panoptic-polarnet: Proposal-free lidar point cloud panoptic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13194-13203, 2021.

Zixiang Zhou, Xiangchen Zhao, Yu Wang, Panqu Wang, and Hassan Foroosh. Centerformer: Center-based transformer for 3d object detection. In European Conference on Computer Vision , pages 496-513, 2022.

Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9939-9948, 2021.

Zhuangwei Zhuang, Rong Li, Kui Jia, Qicheng Wang, Yuanqing Li, and Mingkui Tan. Perception-aware multi-sensor fusion for 3d lidar semantic segmentation. In IEEE/CVF International Conference on Computer Vision , pages 16280-16290, 2021.

Zoox. Zoox autonomous e-shuttle begins operating on public roads in california. https://newatlas.c om/automotive/zoox-electric-robotaxi-public-roads-california , Accessed: 2024-03-06, 2023.

Vlas Zyrianov, Xiyue Zhu, and Shenlong Wang. Learning to generate realistic lidar point clouds. In European Conference on Computer Vision , pages 17-35, 2022.

Appendix

A The Place3D Dataset

In this section, we present additional information on the statistics, features, and implementation details of the proposed Place3D dataset.

A.1 Statistics

Our dataset consists of a total of eleven LiDAR placements, in which seven baselines are inspired by existing self-driving configurations from autonomous vehicle companies, and four LiDAR placements are obtained by optimization. Each LiDAR placement contains four LiDAR sensors. For each LiDAR configuration, the sub-dataset consists of 13,600 frames of samples, comprising 11,200 samples for training and 2,400 samples for validation, following the split ratio used in nuScenes [7]. We combined every 40 frames of samples into one scene, with a time interval of 0.5 seconds between each frame sample.

A.2 Data Collection

In this section, we elaborate on more details of the simulator setup and LiDAR setup procedures used in collecting the Place3D dataset.

A.2.1 Simulator Setup

We choose four maps (Towns 1, 3, 4, and 6, cf. Figure 7) in CARLA v0.9.10 to collect point cloud data and generate ground truth information. For each map, we manually set six ego-vehicle routes to cover all roads with no roads overlapped. The frequency of the simulation is set to 20 Hz.

(a) Town 1

(b) Town 3

(c) Town 4

(d) Town 6

D

A.2.2 LiDAR Setup

LiDAR point cloud data is collected every 0.5 simulator seconds utilizing the Semantic LIDAR sensor in CARLA. The attributes of Semantic LIDAR sensor are listed in Table 5. Notably, we utilize LiDARs with relatively low resolution to increase the challenge in object detection and semantic segmentation, leading to lower scores than those on the nuScenes [7] leaderboard. This is based on the observation that LiDAR point clouds become sparse when detecting distant objects and the use of multiple LiDARs is specifically to enhance the perception of sparse point clouds at long distances in practical applications. Our experiments with low-resolution LiDARs allow for a more evident assessment of the impact of LiDAR placements on the perception of sparse point clouds.

A.3 Label Mappings

In this section, we provide more details for the class definitions of the LiDAR semantic segmentation and 3D object detection tasks.

A.3.1 LiDAR Semantic Segmentation

There are a total of 21 semantic categories in our LiDAR semantic segmentation dataset, which are 1 Building , 2 Fence , 3 Other , 4 Pedestrian , 5 Pole , 6 Road Line , 7 Road , 8 Sidewalk , 9 Vegetation , 10 Vehicle , 11 Wall , 12 Traffic Sign , 13 Ground , 14 Bridge , 15 Rail Track , 16 Guard Rail , 17 Traffic Light , 18 Static , 19 Dynamic , 20 Water , and 21 Terrain . Table 6 presents the detailed definition of each semantic class in Place3D. Additionally, we include a Unlabeled tag to denote elements that have not been categorized.

(a) Waymo

(b) Motional

(c) Cruise

(d) Pony.ai

(e) Zoox

(f) Toyota

(g) Momenta

(h) Ford

cruse

Ford

momenTa

A.3.2 3D Object Detection

We set 3 types of common objects in the traffic scene for the 3D object detection tasks, i.e. , 1 Car , 2 Bicyclist , and 3 Pedestrian , following the same configuration as KITTI [29].

A.4 Adverse Conditions

As shown in Figure 9 and Figure 10, the adverse conditions in the Place3D dataset can be categorized into three distinct groups:

-Severe weather conditions , including 'fog', which causes back-scattering and attenuation of LiDAR points due to water particles in the air; 'snow', where snow particles intersect with laser beams, affecting the beam's reflection and causing potential occlusions; and 'wet ground', where laser pulses lose energy upon hitting wet surfaces, leading to attenuated echoes. These are commonly occurring corruptions in real driving conditions.

-External disturbances like 'motion blur', which results from vehicle movement that causes blurring in the data, especially on bumpy surfaces or during rapid turns.

-Internal sensor failure , such as 'crosstalk', where the light impulses from multiple sensors interfere with each other, creating noisy points; as well as 'incomplete echo', where darkcolored objects result in incomplete LiDAR readings.

A.5 License

The Place3D dataset is released under the CC BY-NC-SA 4.0 license 1 .

B Additional Implementation Detail

In this section, we present additional implementation details to facilitate the reproduction of the data generation, placement optimization, and performance evaluations in this work.

B.1 LiDAR Placement

We adopt five commonly employed heuristic LiDAR placements, which have been adopted by several leading autonomous driving companies (see Figure 8), as our baseline. We create another two baseline placements by adjusting the rolling angles of LiDAR sensors. We obtain two placements in optimization and two placements in the ablation study. All configurations are presented in Table 7.

■

■

■

(a) fog

(b) wet ground

(c) snow

(d) motion blur

(e) crosstalk

(f) incomplete echo

B.2 Corruption Generation

In this work, we adopt the public implementations from Robo3D [46] to generate corrupted point clouds. We follow the default configuration of generating nuScenes-C to construct the adverse condition sets in Place3D. Specifically, for 'fog' generation, the attenuation coefficient is randomly sampled from [0 , 0 . 005 , 0 . 01 , 0 . 02 , 0 . 03 , 0 . 06] and the back-scattering coefficient is set as 0 . 05 . For 'wet ground' generation, the water height is set as 1 . 0 millimeter. For 'snow' generation, the value of snowfall rate parameter is set to 1 . 0 . For 'motion blur' generation, the jittering noise level is set as 0 . 30 . For 'crosstalk' generation, the disturb noise parameter is set to 0 . 07 . For 'incomplete echo' generation, the attenuation parameter is set to 0 . 85 .

B.3 Hyperparameters

In this work, we build the LiDAR placement benchmark using the MMDetection3D codebase [20]. Unless otherwise specified, we follow the conventional setups in MMDetection3D when training and evaluating the LiDAR semantic segmentation and 3D object detection models.

The detailed training configurations of the four LiDAR semantic segmentation models, i.e. , MinkUNet [18], SPVCNN [86], PolarNet [111], and Cylinder3D [117], are presented in Table 8. The detailed training configurations of the four 3D object detection models, i.e. , PointPillars [51], CenterPoint [107], BEVFusion-L [63], and FSTR [109], are presented in Table 9.

(a) fog

(b) wet ground

(c) snow

(d) motion blur

(e) crosstalk

(f) incomplete echo

All LiDAR semantic segmentation models are trained and tested on eight NVIDIA A100 SXM4 80GB GPUs. All 3D object detection models are trained and tested on four NVIDIA RTX 6000 Ada 48GB GPUs. The models are evaluated on the validation sets. We do not include any type of test time augmentation or model ensembling when evaluating the models.

C Additional Quantitative Result

In this section, we provide additional quantitative results to better support the findings and conclusions drawn in the main body of this paper.

C.1 LiDAR Semantic Segmentation

We provide the complete ( i.e. , the class-wise IoU scores) results for LiDAR semantic segmentation under different LiDAR placement strategies.

We showcase the per-class LiDAR semantic segmentation results of MinkUNet [18], SPVCNN [86], PolarNet [111], and Cylinder3D [117] in Table 11. The performance of these methods is evaluated under different LiDAR placement strategies. Different LiDAR placement strategies demonstrated varying propensities towards particular classes. For instance, Pyramid performs well for building , guard rail , and vegetation compared with other placements, possibly due to an increased vertical field of view that captures these taller or layered structures more effectively. Ours provides the generally best balance of performance across categories, with high scores in building , road , and

vehicle suggesting an effective all-around coverage for various object types. The complete benchmark results of LiDAR semantic segmentation under clean and adverse conditions are presented in Table 12. We showcase the correlation between M-SOG and LiDAR semantic segmentation performance under clean and adverse conditions in Figure 11 and 12.

We showcase the complete results of the ablation study for semantic segmentation in Table 14. we first explore the performance of our optimization algorithm under placement constraints, specifically, standardize the elevation of LiDARs to analyze their optimal positioning on the vehicle's roof's horizontal plane. Our optimized placement 2D plane achieved better performance compared with line , square , and trapezoid configurations. Further, we optimize the placement for adverse conditions. While the observation suggests a trade-off in clean data compared with the Ours , the configuration optimized on corruption data ( corruption optimized ) achieves a significantly higher performance in adverse conditions than both baselines in Table 12 and Ours , indicating that custom optimization strategies, tailored for challenging conditions, represent an effective methodology to enhance robust perception capabilities in adverse environments.

C.2 3D Object Detection

In this section, we present more experimental results for the 3D robustness evaluation of 3D object detection under adverse conditions. The performance of PointPillars [51], CenterPoint [107], BEVFusion-L [63], and FSTR [109] is evaluated under different LiDAR placement strategies in Table 13. Across all placements and conditions, there is a trend where all methods suffer a drop in performance in adverse conditions compared to clean conditions, highlighting the challenge that adverse conditions pose to 3D object detection. While some placements, like Pyramid and Square, appear to maintain relatively high performance under both clean and adverse conditions, the Ours placement has the highest average mAP under adverse conditions. In addition, certain placements exhibit relative strengths against specific types of corruption, which can inform the development of more resilient object detection systems for autonomous vehicles. For instance, Pyramid and Pyramid-Roll , appear to handle fog better than others, possibly due to a configuration that captures a more diverse set of angles which could mitigate the scattering effect of fog on LiDAR beams. We showcase the correlation between M-SOG and 3D Object Detection performance under clean and adverse conditions in Figure 13 and 14.

C.3 Compare to Existing Approaches

The effectiveness of sensor placement metrics is determined by the linear correlation between these metrics and actual performance. As shown in Table 10, Jiang's method [40] is applied to roadside placements, while comparable on-vehicle methods, S-MIG [34] and S-MS [57], follow the same process for LiDARs. All these methods use 3D bounding boxes as priors to understand the 3D scene distribution. However, this approach leads to significant deviations from the actual physical boundaries of objects and fails to account for occlusion relationships between objects and the environment in LiDAR applications. Consequently, they cannot accurately describe the scene or effectively optimize LiDAR placements. Moreover, these methods are limited to evaluating LiDAR placements for detection tasks since they solely rely on 3D object distribution information.

Our method addresses these limitations by introducing semantic occupancy information as a prior for the evaluation metric. This allows for more accurate characterization of boundaries in the 3D scene and effectively addresses occlusion issues between objects and the environment. Additionally, our method leverages semantic distribution information under diverse conditions, thereby enhancing the reliability of perception in adverse weather conditions and during sensor failures.

D Additional Qualitative Result

In this section, we provide additional qualitative examples to help visually compare different conditions presented in this work.

D.1 Clean Condition

We showcase the qualitative results of the MinkUNet [18] model using our optimized LiDAR placement. As shown in Figure 15, the depiction across the first four rows demonstrates a commendable performance of our LiDAR placement strategy during clean conditions. This is evidenced by the predominance of gray in the error maps, which indicates a high rate of correct predictions. Also, the fifth row presents a notable exception, illustrating a failure case. This particular scene exhibits a heightened complexity with a diverse array of objects and potential occlusions that challenge predictive accuracy. The qualitative results underscore the importance of continuous refinement of LiDAR placement to achieve better perception across a comprehensive spectrum of scenarios.

D.2 Adverse Conditions

In Figure 16, we showcase the qualitative results of MinkUNet [18] under different adverse conditions utilizing our optimized LiDAR placement. The corruptions from top to bottom rows are 'fog', 'wet ground', 'motion blur', 'crosstalk', and 'incomplete echo'. As can be seen, the LiDAR semantic segmentation model encounters extra difficulties when predicting under such scenarios. Erroneous predictions tend to appear in regions contaminated by different types of noises, such as airborne particles, disturbed laser reflections, and jittering scatters. It becomes apparent that enhancing the model's robustness under these adverse conditions is crucial for the practical usage of driving perception systems. To achieve better robustness, we design suitable LiDAR placements that can mitigate the degradation caused by corruptions. As discussed in Table 12, Table 13, and Table 14, our placements can largely enhance the robustness of various models.

PolarNet

67.2

66.8

67.7

67.7

69.3

70.9

71.0

76.7

64

68

72

76

80

-3.5

-3.0

-2.5

-2.0

-1.5

-1.0

M-SOG(10

-6

)

mIoU (%)

(a) PolarNet

58.5

59.0

59.7

62.7

60.7

62.2

65.7

66.5

56

60

64

68

72

-3.5

-3.0

-2.5

-2.0

-1.5

-1.0

M-SOG(10

-6

)

mIoU (%)

MinkUNet

(b) MinkUNet

60.6

61.0

59.3

67.6

63.4

67.9

67.1

68.3

57

61

65

69

73

-3.5

-3.0

-2.5

-2.0

-1.5

-1.0

M-SOG(10

-6

)

mIoU (%)

SPVCNN

(c) SPVCNN

69.8

68.5

68.9

68.4

69.9

69.3

72.7

73.0

64

68

72

76

-3.5

-3.0

-2.5

-2.0

-1.5

-1.0

M-SOG(10

-6

)

mIoU (%)

Cylinder3D

(d) Cylinder3D

PolarNet

38.2

39.3

40.6

37.2

38.2

36.8

40.4

47.4

34

38

42

46

50

-3.5

-3.0

-2.5

-2.0

-1.5

-1.0

M-SOG(10

-6

)

mIoU (%)

(a) PolarNet

44.1

44.5

47.6

44.4

48.4

45.4

45.0

48.6

40

44

48

52

56

-3.5

-3.0

-2.5

-2.0

-1.5

-1.0

M-SOG(10

-6

)

mIoU (%)

MinkUNet

(b) MinkUNet

43.1

43.8

41.5

46.9

46.6

47.1

43.9

48.6

38

42

46

50

54

-3.5

-3.0

-2.5

-2.0

-1.5

-1.0

M-SOG(10

-6

)

mIoU (%)

SPVCNN

(c) SPVCNN

41.7

42.0

44.8

39.7

43.4

44.5

47.1

47.3

36

40

44

48

52

-3.5

-3.0

-2.5

-2.0

-1.5

-1.0

M-SOG(10

-6

)

mIoU (%)

Cylinder3D

(d) Cylinder3D

43.4

43.8

43.5

44.6

46.1

46.1

46.5

46.8

42

44

46

48

-1.7

-1.6

-1.5

-1.4

-1.3

-1.2

M-SOG(10

-6

)

mAP (%)

PointPillars

(a) PointPillars

49.3

49.2

50.2

50.8

50.7

51.0

52.5

53.0

48

50

52

54

-1.7

-1.6

-1.5

-1.4

-1.3

-1.2

M-SOG(10

-6

)

mAP (%)

BEVFusion-L

(b) BEVFusion-L

54.0

54.0

55.4

55.2

56.2

55.9

55.8

57.1

52

54

56

58

-1.7

-1.6

-1.5

-1.4

-1.3

-1.2

M-SOG(10

-6

)

mAP (%)

CenterPoint

(c) CenterPoint

51.9

52.8

54.6

53.5

55.5

55.7

55.3

56.6

51

53

55

57

59

-1.7

-1.6

-1.5

-1.4

-1.3

-1.2

M-SOG(10

-6

)

mAP (%)

FSTR

(d) FSTR

26.3

27.3

26.1

26.9

28.0

28.9

28.3

28.9

24

26

28

30

32

-1.7

-1.6

-1.5

-1.4

-1.3

-1.2

M-SOG(10

-6

)

mAP (%)

PointPillars

(a) PointPillars

21.9

22.3

25.8

19.5

23.9

24.7

20.8

26.1

18

20

22

24

26

28

-1.7

-1.6

-1.5

-1.4

-1.3

-1.2

M-SOG(10

-6

)

mAP (%)

BEVFusion-L

(b) BEVFusion-L

22.6

24.1

23.2

22.2

28.1

29.4

28.4

29.5

20

22

24

26

28

30

32

-1.7

-1.6

-1.5

-1.4

-1.3

-1.2

M-SOG(10

-6

)

mAP (%)

CenterPoint

(c) CenterPoint

27.7

28.6

28.7

26.6

31.1

30.9

29.2

32.1

24

26

28

30

32

34

-1.7

-1.6

-1.5

-1.4

-1.3

-1.2

M-SOG(10

-6

)

mAP (%)

FSTR

(d) FSTR

Ground Truth

Error Map

Prediction

Dm

Ground Truth

Error Map

Prediction

E Proofs

In this section, we present the full proof of Theorem 1 to certify the global optimally regarding the solved local optima. Additionally, we present the proof of Corollary 1 as a special case with bounded hyper-rectangle search space only given the Lipschitz constant of the objective function.

E.1 Full Proof of Theorem 1

Theorem 2 (Optimality Certification) . Given the continuous objective function G : R n → R with Lipschitz constant k G w.r.t. input u ∈ U ⊂ R n under ℓ 2 norm, suppose over a δ -density Grids subset S ⊂ U , the distance between the maximal and minimal of function G over S is upper-bounded by C M , and the local optima is u ∗ S = arg min u ∈ S G ( x ) , the following optimality certification regarding x ∈ U holds that:

where u ∗ is the global optima over U .

Proof. Based on the sampling over subset S ⊂ U with ℓ 2 -norm density δ , we have:

From the continuous objective function G : R n → R with Lipschitz constant k G w.r.t. input u ∈ U ⊂ R n under ℓ 2 norm, it holds that:

Since the distance between the maximal and minimal of function G over S is upper-bounded by C M , we then have:

Then by absolute value inequality with u S ∈ S ⊂ U s.t. ∥ u ∗ -u S ∥ 2 ≤ δ and combining all inequalities above, we have:

which concludes the proof.

E.2 Proof of Corollary 1

Corollary 2. When U is a hyper-rectangle with the bounded ℓ 2 norm of domain U i ∈ R at each dimension i , with i = 1 , 2 , . . . , n , Thm.2 can hold in a more general way by only assuming that the Lipschitz constant k G of the objective function is given, where the following optimality certification regarding x ∈ U holds that:

Proof. Similar to the proof of Thm. 2, by the sampling over subset S ⊂ U with ℓ 2 -norm density δ , we have:

From the continuous objective function G : R n → R with Lipschitz constant k G w.r.t. input u ∈ U ⊂ R n under ℓ 2 norm, it holds that:

Now since the input space U is a hyper-rectangle with the bounded ℓ 2 norm of domain U i ∈ R at each dimension i , with i = 1 , 2 , . . . , n , for any i -th dimension u ( i ) 1 , u ( i ) 2 ∈ U ( i ) ⊂ R , it holds that:

By summing up all dimensions with absolute value inequality at u max , u min , we have:

Since the distance between the maximal and minimal of function G over S is upper-bounded by C M , we then have:

Then by absolute value inequality with u S ∈ S ⊂ U s.t. ∥ u ∗ -u S ∥ 2 ≤ δ and combining all inequalities above, we have:

which concludes the proof.

F Discussions

In this section, we discuss the limitations and potential negative societal impact of this work.

F.1 Limitations

F.1.1 Possible Limitation in Data Collection

In this work, we utilize CARLA to generate LiDAR point clouds instead of collecting data in the real world. Although the delicacy and realism of simulation technologies are now remarkably close to the real world, there still exists a gap. For example, LiDAR cannot detect transparent objects, and LiDAR generates noise when encountering walls; these characteristics are not well-represented by the simulator. In addition, for each placement, we collected 13,600 frames of data; however, real-world applications would require a significantly larger dataset.

F.1.2 Possible Limitation in LiDAR Configurations

Although we selected seven simplified placements as baselines for extensive experimentation, our baseline placements cannot reflect and cover all possible LiDAR arrangements for various car manufacturers. Furthermore, the optimized placements we obtained are only near-optimal for our dataset. Identifying the globally optimal placements requires further analysis. In addition, our dataset employs spinning LiDAR technology. Yet, the latest advancements in autonomous driving have indicated a trend toward adopting semi-solid-state and solid-state LiDAR systems, necessitating further research. To demonstrate the impact of LiDAR placement, we selected LiDAR sensors with relatively low resolution. The outcomes might differ when the total number of LiDAR sensors is not four or when using LiDAR sensors with different channels and scanning frequencies.

F.1.3 Possible Limitation in Surrogate Metric

When generating the SOG (semantic occupancy grids) for computing the M-SOG metric, we use a voting algorithm to determine the semantic label for each voxel. This approach can introduce potential errors in cases where multiple semantic labels within a voxel have a similar number of points. Additionally, to accurately describe the semantic distribution of a scene, smaller voxel sizes are often required.

F.2 Potential Societal Impact

LiDAR systems, by their very nature, are designed to capture detailed information about the environment. This can include not just the shape and location of objects, but potentially also capturing point cloud data of individuals, vehicles, and private property in high resolution. The data captured might be misused if not properly regulated and secured. Further, With autonomous vehicles navigating spaces using LiDAR and other sensors, there could be a shift in how spaces are designed, potentially prioritizing efficiency for autonomous vehicles over human-centered design principles. In addition, over-reliance on LiDAR and perception systems can lead to questions about the trustworthiness and reliability of these systems.

G Public Resources Used

We acknowledge the use of the following public resources, during the course of this work:

14

https://github.com/xinge008/Cylinder3D

.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made in the abstract and introduction sections have been phrased to reflect our contributions and scope.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The potential limitations and societal impact are discussed in Section F.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Full sets of assumptions and a complete proof are included. Please refer to Section E.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have fully disclosed all the information needed to reproduce the experimental results of this paper.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We attached the code for our implementations and for generating the training and evaluation data in the attachment.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: This paper has specified all the training and test details. Please refer to Section 4 and Section B.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Not applicable.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Sufficient information on the computing resources used in this work has been clearly discussed. Please refer to Section 4 and Section B.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The conducted research sticks with the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discussed these societal impacts in the Appendix. Please refer to Section F.2.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We foresee our paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cited and listed all existing assets in the Appendix. Please refer to

Section G.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We uploaded the code as supplementary material with a clear readme file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This study does not involve crowdsourcing or research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This study does not involve crowdsourcing or research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Place3D, a full-cycle pipeline that encompasses LiDAR placement optimization, data generation, and downstream evaluations. The framework makes three appealing contributions. 1) To identify the most effective configurations for multi-LiDAR systems, this paper introduces a Surrogate Metric of the Semantic Occupancy Grids (M-SOG) to evaluate LiDAR placement quality. 2) Leveraging the M-SOG metric, this paper propose a novel optimization strategy to refine multi-LiDAR placements. 3) Centered around the theme of multi-condition multi LiDAR perception, the authors collect a 364,000-frame dataset from both clean and adverse conditions. Extensive experiments demonstrate that LiDAR placements optimized using palce3D outperform various baselines.\n\nSTRENGTHS:\n- This paper first attempt at investigating the impact of multi-LiDAR placements for 3D semantic scene, which is seldom explored in LiDAR-based detection and segmenation areas.\n- This paper proposes a more comprehensive metric M-SOG. Experiments show M-SOG is more relevant to the model performance than S-MIG, demonstrating its effectiveness to represent the scene coverage.\n- Guided by M-SOG, a LiDAR placement optimization methods is proposed in this work which can provide a solid baseline for LiDAR placement optimization.\n\nWEAKNESSES:\n- The representation may be a little confused. The M-SOG and the optimization method, under my understand, is task-independent. However, there are two optimized LiDAR placement related to Det and Seg tasks, which is a little confused.\n- To demonstrate the effectiveness of M-SOG, it's recommended to draw a scatter plot of M-SOG and model performance in Tab2 and Tab3, which can be more intuitive.\n\nQUESTIONS:\n- I'm curious about the influence of the LiDAR numbers. It's better to add a experiment to explore the most cost-effective LiDAR number.\n- M-SOG seems to be a global metric. Can It also reflect the scene coverage of a specific local area?\n\nLIMITATIONS:\nNan", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper targets an overlooked but important aspect of 3D scene understanding -- the influence of LiDAR sensor placements for the success of 3D perception tasks, such as 3D detection and 3D segmentation.\n\nThe authors introduce Place3D, a comprehensive pipeline for optimizing LiDAR placement in autonomous driving scenarios to enhance 3D scene understanding, especially under some adverse conditions. \n\nIndeed, current single-LiDAR systems and datasets tend to fail to capture the complexities of real-world environments. This work proposes to address this by using the Surrogate Metric of the Semantic Occupancy Grids (M-SOG) to evaluate LiDAR placement quality. M-SOG measures the information gain and scene understanding capability by calculating the entropy of semantic voxel distributions.\n\nThe optimization strategy employs the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to iteratively improve LiDAR configurations. The objective function is the M-SOG score, ensuring that the optimized placements are close to the global optimum. Compared to the existing S-MIG, this work makes a significant contribution by extending the binomial occupancy distribution to multinomial occupancy distribution through multiple semantic labels for better scene understanding.\n\nPlace3D also introduces a dataset of 364,000 frames collected under various conditions, including severe weather and sensor failures. The optimized placements show superior performance in LiDAR semantic segmentation and 3D object detection tasks compared to existing baselines.\n\nSTRENGTHS:\n1. This work focuses on a less-explored but rather important perspective of 3D scene understanding. The established LiDAR placement evaluation benchmark and the proposed placement optimization method could open up some new and interesting research directions in related research areas.\n\n2. An innovative metric,  M-SOG, for evaluating LiDAR placement quality by incorporating semantic information, which improves accuracy over previous methods. Additionally, this work contributes a full-cycle approach from LiDAR placement optimization to data generation and downstream evaluation.\n\n3. The proposed Place3D dataset includes several adverse conditions for realistic and practical evaluation, demonstrating significant improvements in robustness. Additionally, the use of CMA-ES ensures that LiDAR placements are close to the global optimum, with strong theoretical optimality certification.\n\nWEAKNESSES:\n1. In the experiments, although the baseline LiDAR placement methods are based on existing configuration of current autonomous vehicle systems (Fig. 7), it is better to compare the proposed method with some randomly sampled LiDAR placements. This can further justify the effectiveness of CMA-ES compared to a wider range of scenarios.\n\n2. It is not clear how the density of the occupancy grid (Sec. 3.3) will influence the surrogate metric, as the estimation of semantic distribution depends on the number of samples.\n\n3. The experiments are conducted in the simulation. Although CARLA is a rather realistic simulator and has been widely adopted in academia, it is unknown how well the proposed method works in the real-world setting. Perhaps it is not feasible to conduct on-site experiments, but more justifications regarding this aspect are needed for further discussion. \n\n4. The optimization process and data generation might require substantial computational resources. Either the main body or the appendix omitted such details. It is highly recommended to supplement such details as they are critical for follow-up works.\n\n5. Can the proposed benchmarks and optimization strategies be extended to other 3D scene understanding tasks, such as 3D object tracking, semantic occupancy prediction, etc?\n\nQUESTIONS:\nPlease refer to the above section for detailed questions.\n\nLIMITATIONS:\nThe authors have included some discussions of the potential limitations and societal impact in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a very interesting study of optimal lidar placement on autonomous vehicles. They introduce a novel measure of the quality of lidar placement called M-SOG, and also discuss an optimization approach to find the best placement. They measure the impact of the placement on the important vision tasks in autonomous vehicles, such as object detection and semantic segmentation, showing that the placement metric is aligned with the improved quality. Lastly, in addition to clear weather they also analyze the impact on data when the conditions are degraded (such as fog or faulty sensors). Moreover, they also provide the data and the code that supports their work.\n\nSTRENGTHS:\n- Very interesting and timely work.\n- The new metric and the results could be of interest to the self-driving community, and the methods should not be difficult to implement.\n- The presented results are quite promising.\n\nWEAKNESSES:\n- The paper writing is can be improved significantly, as the method is not actually clearly explained.\n- The authors tried to stuff too many things in a single paper (metric and degraded sensors), which degraded focus and impact of the work.\n- The experiments can be improved significantly.\n(detailed comments on the above weaknesses can be found in the section below)\n\nQUESTIONS:\n- Line 107, \"voxelized grips\" -> \"voxels\"?\n- Line 110, \"where each voxel is assumed to be independently and identically distributed\", what does this actually mean? The voxels are clearly not independent of each other.\n- The notation can be improved, as it is confusing at times and not properly introduced. E.g., line 112, what is T, 1.\n- Along similar lines, H is not properly introduced a few lines below.\n- Line 143, should P_SOG also have k-index in its name? As it is defined for a particular k, as per the given equation.\n- In the paragraph starting with line 150, the authors discuss the metric. However, given that the metric is computed using only a subset of voxels, is there an invalid setup that the metric would not capture properly, such as putting lidars below the car or something similar?\n- Fig 2, would be good to zoom in on the roof of the car to make it more visible, as that is the relevant part in this figure.\n- The optimization in Section 3.3 is not well explained, and the notation is also not properly introduced.\n- E.g., the authors do not properly explain what is the lidar configuration space, or what are the constraints. Some details can be gleaned here and there, but given how critical this part is for their work the authors should do a much better job explaining it.\n- Along these lines, line 173 mentions cov-mat of distribution, distribution of what? This is very poorly explained.\n- Also, in this paragraph the authors use k-index for samples, but k was already used for classes?\n- How significant and how practical is the theoretical analysis? The authors introduce C_M as a gap, but this constant is not discuss anywhere, or how large it is. This makes the theoretical discussion quite weak. More discussion and more clarifications are needed.\n- Line 238, Tables 2 and 3 don't actually show this.\n- The detectors used in these experiments are not clearly discussed.\n- In general, it seems that the authors tried to stuff too many things at the same time in the work and in the experiments. This made them relegate many results and discussions to the appendix, even those that are critical to the work. It seems that it would be much better to focus on one aspect first (e.g., metric + impact on vision results in clear weather) and only then expand from there.\n- What are the points in Fig 4, not clear and not discussed.\n- The adverse aspect is very important, yet the authors do not spend a lot of time/space on it, and the results are just given. This is unfortunate, and weakens their argument around this part of their work.\n- Line 253, this was already mentioned in line 238?\n- Line 263, why is the placement good for adverse weather? Any deeper insights? How generalizable are these placements? All these important questions are not discussed at all, and the authors just skip them. This is another symptom of stuffing too many things in one paper and losing focus and depth.\n- Line 270, the authors just move all important ablation results to the appendix, which is not acceptable. At this point the authors are just abusing the appendix. Please note that the appendix is not meant to increase the page limit, and critical experiments/results/discussions should NOT be moved there.\n- More discussions and insights about the found placement should be added, such as around 2D or roll solution. Any advice for the designers, any interesting findings? This is not provided at all.\n\n###################\nFEEDBACK AFTER THE REBUTTAL\n###################\n\nI would like to thank the authors for responding to my comments, and also to the other reviewers who helped me further understand the paper. However, after reading through all the other responses/discussion it seems that my main concerns were not addressed. E.g.,while I said that the paper is too stuffed and poorly explained, the authors just said that they will add more explanations and that's that. First, I'm not sure where would this be added since there is no space and the paper is overstretched as is, and second, the required changes are significant and just saying that it will be addressed is not enough, the paper would need to be re-reviewed. Moreover, the authors marked my comments under \"Other Minor Modifications\", which just shows that they did not appreciate the severity of my comments.\nI do like the idea and I do think that the work has future, but with the current execution I just can not see how can I increase my recommendation. The paper needs quite a lot of attention and rewriting, so much so that I am not confident that the authors would not be able to do that with quick updates, especially given their responses that made light of some of my more important comments.\n\nLIMITATIONS:\nThe authors do not discuss the limitations in a separate section.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework called Place3D to investigate the placement of multiple LiDAR sensors for semantic segmentation and object detection task under various weather conditions.  Place3D mainly introduce a Surrogate Metric of Semantic Occupancy Grids (M-SOG) to evaluate the perception performance with different sensors placements.  Besides, an optimization approach is applied to refine the LiDAR placement. The method is evaluated on the simulated dataset with CARLA, showing the advantages of the refined placements.\n\nSTRENGTHS:\n- The motivation is interesting and the paper is well-written.\n- A new metric and a dataset are proposed for the evaluation of the placement. \n- The experiments are extensive, three typical models are evaluated on 7 weather scenarios. \n- Comparison with 7 popular placement methods show the advantages of the refined placement with the proposed methods.\n\nWEAKNESSES:\n- Although the proposed method  investigates LiDAR placement for more tasks under more weathers, the core formulation is similar to that in [34] as cited in the paper.\n- Only one type of 16-beam LiDAR sensor is investigated. Actually, sensors with more laser beams (64 or even 128) are becoming low-cost and popular. It is important to evaluate the effect of the placement as the point cloud data get dense.\n- During the companions in Tab.2,3, different refined placements are applied for different tasks. How is the performance if applied a Seg-optimized placement to the Det task? It is impractical to change the placement for different task. Is that possible to find an optimal placement for both tasks?\n\nQUESTIONS:\n- In the dataset comparison part, it is written that no adverse weather for Nuscenes and Waymo.  They may include rainy weather and even more (e.g., Snow).  Although in the appendix part, please also have a check on that.\n- How is the solution spaces for the LiDAR placement? Are the displacements along z-axis or even rotation are considered during the optimization?\n- Is there any intuitive interpretation about why refined sensor placement are beneficial to different tasks?\n\nLIMITATIONS:\nThe authors discussed the possible limitation in data collection.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}