Is Your LiDAR Placement Optimized for 3D Scene Understanding?

Authors: ye  li
lingdong  kong
hanjiang  hu
xiaohao  xu
xiaonan  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 06:17:18

Abstract: The reliability of driving perception systems under unprecedented conditions is crucial for practical usage. Latest advancements have prompted increasing interest in multi-LiDAR perception. However, prevailing driving datasets predominantly utilize single-LiDAR systems and collect data devoid of adverse conditions, failing to capture the complexities of real-world environments accurately. Addressing these gaps, we proposed Place3D, a full-cycle pipeline that encompasses LiDAR placement optimization, data generation, and downstream evaluations. Our framework makes three appealing contributions. 1) To identify the most effective configurations for multi-LiDAR systems, we introduce the Surrogate Metric of the Semantic Occupancy Grids (M-SOG) to evaluate LiDAR placement quality. 2) Leveraging the M-SOG metric, we propose a novel optimization strategy to refine multi-LiDAR placements. 3) Centered around the theme of multi-condition multi-LiDAR perception, we collect a 280,000-frame dataset from both clean and adverse conditions. Extensive experiments demonstrate that LiDAR placements optimized using our approach outperform various baselines. We showcase exceptional results in both LiDAR semantic segmentation and 3D object detection tasks, under diverse weather and sensor failure conditions.

Main paper: 
 introduction: 
 Accurate 3D perception plays a crucial role in autonomous driving, involving detecting the objects around the vehicle and segmenting the scene into meaningful semantic categories. LiDARs are becoming crucial for driving perception due to their capability to capture detailed geometric information about the surroundings [3, 109, 53, 54, 118] . While the latest models achieved promising accuracy on standard datasets, e.g., nuScenes [7] and SemanticKITTI [6] , improving the resilience of perception under corruptions and sensor failures is still a critical yet under-explored task [46, 94, 5, 95] .Recent studies have primarily focused on refining the sensing systems by designing new algorithms with novel model architectures [15] or 3D representations [28, 99, 11] . The selection of LiDAR configurations, however, often relies on industry experience and design aesthetics. Therefore, existing literature has potentially overlooked optimal LiDAR placements for maximum sensing efficacy.An intuitive approach for optimizing LiDAR configurations involves a comprehensive cycle of data collection, model training, and validation across various LiDAR setups to enhance the autonomous driving system's perception accuracy [24, 52] . However, this approach faces significant challenges due to the substantial computational resources and extensive time required for data collection and processing [57, 17] . Although recent works [34, 65] made preliminary attempts to explore the impact of LiDAR placements on perception accuracy, they have neither proposed an optimization method nor evaluated the performance in adverse conditions.In this work, we delve into the optimization of sensor configurations for autonomous vehicles by tackling two critical sub-problems: 1) the performance evaluation of sensor-specific configurations, and 2) the optimization of these configurations for enhanced 3D perception tasks, encompassing 3D object detection and LiDAR semantic segmentation. To achieve this goal, we propose a systematic LiDAR placements evaluation and optimization framework, dubbed Place3D. The overall pipeline is endowed with the capability to synthesize point cloud data with customizable configurations and diverse conditions, including common corruptions, external disturbances, and sensor failures.We first introduce an easy-to-compute Surrogate Metric of Semantic Occupancy Grids (M-SOG) to evaluate the equality of LiDAR placements. Next, we propose a novel optimization approach utilizing our surrogate metric based on Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to find the near-optimal LiDAR placements. To verify the correlation between our surrogate metric and assess the effectiveness of our optimization approach on both clean and adverse conditions, we design an automated multi-condition multi-LiDAR data simulation platform and establish a comprehensive benchmark consisting of a total of seven LiDAR placement baselines inspired by existing self-driving configuration from the autonomous vehicle companies.Our benchmark, along with a large-scale multi-condition multi-LiDAR perception dataset, encompasses state-of-the-art learning-based perception models for 3D object detection [51, 107, 63, 109] and LiDAR semantic segmentation [18, 111, 86, 117, 50] , as well as six distinct adverse conditions coped with weather and sensor failures [46, 95] . Utilizing the proposed framework, we explored how various perturbations and downstream 3D perception tasks affect optimization outcomes.To summarize, this work makes the following key contributions:-To the best of our knowledge, Place3D serves as the first attempt at investigating the impact of multi-LiDAR placements for 3D semantic scene understanding in diverse conditions. -We introduce M-SOG, an innovative surrogate metric to effectively evaluate the quality of LiDAR placements for both detection (sparse 3D) and segmentation (dense 3D) tasks. -We propose a novel optimization approach utilizing our surrogate metric to refine LiDAR placements, which exhibit excellent LiDAR semantic segmentation and 3D object detection accuracy and robustness, outperforming baselines for relatively 9% in metrics. -We contribute a 280,000-frame multi-condition multi-LiDAR point cloud dataset and establish a comprehensive benchmark for LiDAR-based 3D scene understanding evaluations. We hope this work can lay a solid foundation for future research in this relevant field. 
related work: 
 LiDAR Sensing. LiDAR sensing is critical in autonomous vehicles, providing essential structural information for their operation [59, 3, 73] . Utilizing 3D data, LiDAR supports tasks such as semantic scene understanding [7, 26, 29, 6, 85, 48, 10] , generation [75, 120, 74, 97] , decision making [25, 8, 19, 87] , and simulation [23, 68, 67, 93] . This work specifically targets LiDAR-based perception and simulation, which are at the forefront of current research in autonomous vehicle technology.LiDAR-Based 3D Scene Understanding. LiDAR segmentation and 3D object detection are key tasks for 3D scene understanding. Segmentation models are categorized into point-based [88, 35, 36, 110, 77] , range view [70, 90, 21, 98, 113, 16, 47, 2, 45, 99] , bird's eye view [111, 115, 12] , voxel-based [18, 86, 117, 33] , and multi-view fusion [58, 38, 78, 60, 62, 14, 13, 61, 100] methods. While they show promising results on benchmarks, their performance across different LiDAR configurations is not well explored. For 3D object detection, models typically use point-based [82, 106, 105, 116] or voxel-based [114, 51, 69, 92, 83, 55, 66, 104] representations, with recent works favoring fusion for improved detection [64, 80, 81, 63, 56] . Our study complements these approaches by optimizing LiDAR placements to enhance performance under various real-world conditions.LiDAR Perception Robustness. The reliability of LiDAR-based 3D scene understanding models in real-world scenarios is crucial [84, 39] . Recent studies have explored the robustness of 3D perception models against adversarial attacks [112, 96] , common corruptions [46, 49, 94, 103, 43, 4] , adverse weather [31, 30, 79, 37] , sensor failures [108, 15, 28] , and combined motion and sensor perturbations [ 102, 95] . To our knowledge, no prior work has focused on optimizing LiDAR placement to improve semantic 3D scene understanding robustness. Our Place3D addresses this gap by studying various placement strategies to enhance robustness in both in-domain and out-of-domain scenarios.Sensor Placement Optimization. Optimizing sensor placements can enhance performance and address the challenges of heuristic design [42, 101] . In autonomous driving, optimizing LiDAR placements is relatively new [65] . Hu et al. [34] studied multi-LiDAR placements for 3D object detection, while Li et al. [57] examined LiDAR-camera configurations for multi-modal detection. Other works [41, 44, 9, 40] explored roadside LiDAR placements for V2X applications. Distinguishing from these efforts, our work is the first to investigate LiDAR placements for robust 3D scene understanding in challenging conditions. We establish benchmarks for both LiDAR semantic segmentation and 3D object detection, providing an in-depth analysis of optimal placements for robust perception.3 Place3D: A Full-Cycle LiDAR Placement PipelineIn this section, we introduce the Surrogate Metric of Semantic Occupancy Grids (M-SOG) to assess the 3D perception performance of sensor configurations (Section 3.2). Leveraging M-SOG, we propose a novel optimization strategy for multi-LiDAR placement (Section 3.3). Our approach is theoretically grounded; we provide an optimality certification to ensure that optimized placements are close to the global optimum (Section 3.4). Figure 1 depicts an overview of our Place3D framework. 
preliminary formulation of surrogate metric: 
 Region of Interest. Following the literature of sensor placement [34, 57] , we define the Region of Interest (ROI) for perception as a finite 3D cuboid space [l, w, h] with ego-vehicle in the center, and divide the ROI space S into voxels with resolution [δ l , δ w , δ h ] as S = {v 1 , v 2 , ..., v N }, where N = l δ l × w δw × h δ h denotes the total number of voxels. Probabilistic Occupancy Grids (POG). We define POG as the joint probability of all non-zero voxels in the ROI asp P OG = p(v 1 , v 2 , ..., v N ) = N i=1, p(vi)̸ =0 p(v i ),where each voxel is assumed to be independently and identically distributed. The voxel v i is occupied if it is within any 3D bounding box b c of object class c at frame t, which is denoted asv i = b (t)c . Therefore, the probability of being occupied for voxel v i among all T frames can be estimated as p(v i ) = T t=1 1(v i = b (t)c )/T . Probabilistic Surrogate Metric. Now given some LiDAR placement L, the conditional POG can be found as the conditional joint distribution p P OG|L = p(v 1 , v 2 , ..., v N |L) in the literature [34] .To show how much uncertainty the placement L can reduce in the ROI, the naive surrogate metric S-MIG [34] is introduced as the reduction of the entropy as-H P OG|L = - N i=1 H(v i |L) with con- stant H P OG = N i=1 H(v i ),where H(v i ), H(v i |L) can be found through the entropy of Bernoulli distribution p(v i ), p(v i |L), respectively.The naive surrogate metric S-MIG [34] employs 3D bounding boxes as priors to understand 3D scene distribution. However, this approach exhibits substantial deviations from the actual physical boundaries of the objects and overlooks the occlusion relationships between objects and the environment in LiDAR applications. Furthermore, it is limited to detection tasks only. To overcome these limitations, we propose the Surrogate Metric of Semantic Occupancy Grids to assess and optimize LiDAR placement for 3D scene understanding tasks. Leveraging the Semantic Occupancy Grids derived from diverse environments, our approach can be expanded to tackling adverse conditions.Semantic Occupancy Grids (SOG). Given a 3D driving scene with a set of semantic classes Y = {y 1 , y 2 , ..., y M }, where M represents the total number of semantic classes in the scene and assuming that each voxel can only be occupied by one semantic tag for each frame, we denote y (t) (v i ) as the semantic class of voxel v i ∈ {v 1 , v 2 , ..., v N } at time frame t ∈ {1, 2, ..., T }. Notably, empty voxels are also considered a semantic class. Accordingly, the set of voxels occupied by semantic class y c at frame t is defined asS (t) yc = {v i |y (t) (v i ) = y c }, c = 1, 2, ..., M, t = 1, 2, ..., T . Then, we introduce SOG to describe the total semantic voxel distribution:EQUATIONProbabilistic Semantic Occupancy Grids. In contrast to the Bernoulli distribution POG [34] , we propose a more accurate Multinomial distribution: Probabilistic Semantic Occupancy Grids (P-SOG), denoted as p SOG , to represent the probability distribution of voxels belonging to certain semantic classes. Before estimating p SOG , we first traverse all frames from given scenes to obtain the probability p for each voxel v i occupied by each semantic class y c :EQUATIONNotably, we denote voxel v i being occupied by semantic class y c as v i = y c , and p indicates estimated distributions from observed samples, whereas p refers to the statistical parameters to be estimated, which are unknown and non-random. We compute the joint probability of all non-zero voxels in the ROI to estimate the p SOG . Following the literature [34] , the joint distribution over the voxel set S is:EQUATIONThe uncertainty of Probabilistic Semantic Occupancy Grids reflects the 3D scene understanding capability of sensors within a given scene. To quantify this uncertainty, we calculate the entropy for the probability distribution of each voxel v i in pSOG as:Ĥ(v i ) = - M c=1 p(v i = y c ) log p(v i = y c ).Based on the independent and identically distributed assumption of SOG, the entropy of P-SOG over the voxel set S is given by: ĤSOG = E vi∼p S N i=1 Ĥ(v i ), where p S denotes the p SOG over the voxel set S. From the perspective of density estimation, the true P-SOG and its corresponding entropy can be estimated as p SOG = pSOG and H SOG = ĤSOG , respectively. Surrogate Metric of Semantic Occupancy Grids. To evaluate LiDAR placements, we further analyze the joint probability distribution of voxels covered by LiDAR rays with varied LiDAR placements. Leveraging the Amanatides and Woo's Fast Voxel Traversal Algorithm [1] , we obtain the voxels covered by LiDAR rays asS|L j = FastVT(S, L j ) = {v Lj 1 , v Lj 2 , ..., v Lj Nj } given LiDAR configuration L = L j , j = 1,2, ..., J, where j indexes the LiDAR placements, J is the number of total configurations, and N j is the number of voxels covered by rays of LiDAR configuration L = L j . Then, the semantic entropy distribution of P-SOG over the voxel set S|L j can be estimated as:EQUATIONAlgorithm 1 Multi-LiDAR Placement Optimization for Semantic 3D Scene Understanding 1: Initialize: k ← 0, m (0) , σ (0) , C (0) , N k , M k , ∀k ∈ {0, 1, 2, . . . , K} 2: for k = 0, 1, 2, . . . , K do 3:for i = 1 to N k do 4: Sample u (k) i ∼ N (m (k) , (σ (k) ) 2 C (k)) from δ-density gird-level candidates 5:Calculate G(u(k) i ) 6:end for Update σ (k+1) and C (k+1) via Equation ( 7), ( 8), (9) , and (10) 9: end for We further define the information gain of 3D scene understanding as ∆H = H SOG -H Lj SOG to describe the perception capability of LiDAR configuration L j . Since H SOG is a constant given certain 3D semantic scenes with the fixed P-SOG distribution, we propose the normalized Surrogate Metric of Semantic Occupancy Grids (M-SOG) as follows to evaluate the perception capability:EQUATIONWe adopt the heuristic optimization based on the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [32] to find an optimized LiDAR configuration.Objective Function. We define the objective as F (u j ) = M SOG (L j ), where u j ∈ U ⊂ R n represents the LiDAR configuration L j and is subject to the physical constraint P (u j ) = 0, and P (u) > 0 if it is violated, e.g., mutual distance between LiDARs and distance from a 2D plane.In our optimization, u j includes the 3D coordinates and rolling angles of LiDARs, U is the finite cuboid space above the vehicle roof. Without ambiguity, we omit subscript j in the following text. We then transform the constrained optimization into the unconstrained Lagrangian form as G(u) = -F (u)+λP (u), where λ is the Lagrange multiplier. We optimize G(u) through an iterative process that adapts the distribution of candidate solutions as Algorithm 1.Optimization Approach. We first define a multivariate normal distribution N (m (k) , (σ(k) ) 2 C (k) ),where m (k) , σ (k) , and C (k) are the mean vector, step size, and covariance matrix of the distribution of iteration k, respectively. We discretize the configuration space U with density δ and sampleN k candidates u (k) i ∼ N (m (k) , (σ (k) ) 2 C (k)) in each iteration k, where i indexes the candidates. We update mean vector m k+1 for the next iteration k + 1 as the updated center of the search distribution for LiDAR configuration. The overall process can be depicted as follows:EQUATIONwhere M k is the number of best solutions we adopt to generate m (k+1) , and w i are the weights based on solution fitness. We obtain the evolution path p (k+1) Cthat accumulates information about the direction of successful steps as follows:EQUATIONwhere c C is the learning rate for the covariance matrix update. The covariance matrix C controls the shape and orientation of the search distribution for LiDAR configurations. It can be updated at each iteration k in the following format:EQUATIONSimilarly, we update p σ as the evolution path for step size adaptation. Then, the global step size σ can be found below for the scale of search to balance exploration and exploitation:EQUATIONEQUATIONwhere c σ is the learning rate for updating the evolution path p σ . d σ is a normalization factor to calibrate the pace at which the global step size is adjusted.Once the evolution optimization empirically converges, it holds that the optimized solution is the local optima of the δ-density Grids space of LiDAR configuration space. In this section, we provide a stronger optimality certification to theoretically ensure that the optimized placement is close to the global optimum under the assumption of bounded and smooth objective function. Due to space limits, the full proof has been attached to the Appendix Section E. Theorem 1 (Optimality Certification). Given the continuous objective function G : R n → R with Lipschitz constant k G w.r.t. input u ∈ U ⊂ R n under ℓ 2 norm, suppose over a δ-density Grids subset S ⊂ U, the distance between the maximal and minimal of function G over S is upper-bounded by C M , and the local optima is u * S = arg min u∈S G(x), the following optimality certification regarding x ∈ U holds that:EQUATIONwhere u * is the global optima over U.The global optimality certification Theorem 1 is applicable in practice because the Lipschitz constant k G and the distance between the maximal and minimal of objective function G over S can be approximated easily through calculating G(u(k) i ) of Algorithm 1 for each sampled u (k) iover the δdensity Grids subset. Besides, we have a more general corollary below to further relax the assumption of bounded objective function. Corollary 1. When U is a hyper-rectangle with the bounded ℓ 2 norm of domain U i ∈ R at each dimension i, with i = 1, 2, . . . , n, Theorem 1 can hold in a more general way by only assuming that the Lipschitz constant k G of the objective function is given, where Equation ( 11) becomes: Table 1 : Optimized M-SOG Metrics. We report the M SOG scores for both detection and segmentation tasks in our pipeline. The scores are calculated based on the car class for detection and all semantic classes for segmentation. Line-roll and Pyramid-roll are abbreviated as L-roll and P-roll. Data Generation. We generate LiDAR point clouds and ground truth using CARLA [23] . We use the maps of Towns 1, 3, 4, and 6 and set 6 ego-vehicle routes for each map. We incorporate 23 semantic classes for LiDAR semantic segmentation and 3 instance classes for 3D object detection. Data collection is performed for 10 LiDAR placements, resulting in a total of 280,000 frames: 1) For each placement, we gather 340 clean scenes and 360 corrupted scenes, with each scene consisting of 40 frames.EQUATION2) The clean set comprises 13,600 frames, including 11,200 samples (280 scenes) for training and 2,400 samples (60 scenes) for validation, following the split ratio used in nuScenes [7] .3) The corruption set is used for robustness evaluation and contains 6 different conditions, each with 2,400 samples (60 scenes). More details are in the Appendix Section A.LiDAR Configurations. We adopt five commonly employed heuristic LiDAR placements, which have been adopted by leading autonomous driving companies, as our baseline. These placements are represented in Figure 2a -e. Following KITTI [29] , we configured the LiDAR sensor with a vertical field of view of [-24.8, 2.0] degrees. To achieve the Line-roll and Pyramid-roll configurations, we adjusted LiDAR roll angles for the Line and Pyramid setups, as depicted in Figure 2 . We present detailed LiDAR configurations in the Appendix Section B.1.Corruption Types. To replicate adverse conditions, we synthesized six types of corrupted point clouds on the validation set of each sub-set, following the settings of the Robo3D benchmark [46] . These corruptions can be categorized into 1) severe weather conditions, including "snow", "fog", and "wet ground", 2) external disturbances, including "motion blur", and 3) internal sensor failures, including "crosstalk" and "incomplete echo". We include the features and implementation details of corruptions in the Appendix Section B.2.P-SOG Synthesis. We follow the steps as depicted in Figure 3 to generate semantic occupancy grids for each LiDAR scene. We first collect point clouds and semantic labels using high-resolution LiDARs and sequentially divide all samples into multiple subsets. Through the transformation of world coordinates of the ego-vehicle, frames of point clouds of each subset are aggregated into one frame of dense point cloud. Then, we utilize the voting strategy to determine the semantic label for each voxel in ROI and generate the SOG. This process is executed across all subsets to produce P-SOG. Notably, the P-SOG is only generated on the scenes of the training set.Surrogate Metric of SOG. We compute the scores of M-SOG separately for 3D detection and segmentation, as shown in Table 1 . To evaluate M-SOG for segmentation, we utilize all semantic classes to generate semantic occupancy grids. For detection, we specifically focus on the car semantic type, while merging the remaining semantic classes into a single category for M-SOG analysis. Detector and Segmentor. For the benchmark, we conduct experiments using four LiDAR semantic segmentation models, i.e., MinkUNet [18] , SPVCNN [86] , PolarNet [111] , and Cylinder3D [117] , and four 3D object detection models, i.e., PointPillars [51] , CenterPoint [107] , BEVFusion-L [63] , and FSTR [109] . The detailed training configurations are included in the Appendix Section B.3. 
comparative study: 
 We conduct benchmark studies to evaluate the performance of varied LiDAR placements in both clean and adverse conditions. We extensively examine the correlation between the proposed surrogate metric, known as M-SOG, and the final performance results. Through our analysis, we are able to demonstrate the effectiveness and robustness of the entire Place3D framework.Effectiveness of M-SOG Surrogate Metric in Place3D. In Table 1 , we report the scores of the M-SOG surrogate metric for various LiDAR placements. In Tables 2 and 3 , we benchmark the 3D object detection and LiDAR semantic segmentation performance of varied LiDAR placements with state-of-the-art algorithms. Figure 4 illustrates the correlation between the proposed M-SOG Figure 4 : The correlation between M-SOG and LiDAR semantic segmentation [18, 86, 111, 117] models performance in the clean condition. Figure 5 : Comparisons of M-SOG with S-MIG [34] using BEVFusion-L [63] and PointPillars [51] .surrogate metric and perception capacity. The results demonstrate a clear correlation, where the performance generally improves as the M-SOG increases. While there might be fluctuations in some placements with specific algorithms, the overall relationship follows a linear correlation, highlighting the effectiveness of our M-SOG for computationally efficient sensor optimization purposes. We show more plots in the Appendix Sections C.1 and C.2.Comparisons to Existing Sensor Placement Methods. The effectiveness of LiDAR sensor placement metrics lies in the linear correlation between metrics and actual performance. We set the same ROI size and conduct a quantitative comparison with S-MIG [34] on 3D detection (as S-MIG [34] only works for 3D detection task). As shown in Figure 5 , our metric exhibits better linear correlation.Since we introduce semantic occupancy information as a prior for the evaluation metric, our method more accurately characterizes the boundaries in the 3D semantic scene and addresses the issue of objects and environment occlusions when using 3D bounding boxes as priors. Moreover, our LiDAR placement method makes the first attempt to include both detection and segmentation tasks. We present additional comparisons in the Appendix Section C.3.Superiority of Optimization via Place3D. As shown in Tables 2 and 3 , our optimized configurations achieved the best performance in both segmentation and detection tasks under both clean and adverse conditions among all models. We report per-class quantitative results in the Appendix Sections C.1 and C.2. The improvement from optimization remains significant even when comparing our optimized configurations against the best-performing baseline in each task. For segmentation, optimized placements outperform the best-performing baseline by up to 0.8% on clean datasets and by as much as 1.5% on corruption. For detection, optimized configurations exceed the best-performing baseline by up to 0.7% on clean datasets and by up to 0.3% on corruption.Robustness of Optimization via Place3D. The M-SOG utilizes semantic occupancy as prior knowledge, which is invariant to changing conditions, thereby enabling robustness of optimized placement in adverse weather. Under the corrupted setting, although the correlation between M-SOG and perception performance is not as obvious as that in the clean setting and shows some fluctuation, our optimized LiDAR configuration consistently maintained its performance in adverse conditions, mirroring its effectiveness in the clean condition. While the top-performing baseline LiDAR configurations in clean datasets might be notably worse compared to others when faced with corruption, the optimized configuration via Place3D consistently shows the best performance under adverse conditions. We showcase several qualitative results on this aspect in the Appendix Section D.Intuitive Interpretation. We observe several intuitive reasons why refined sensor placement is beneficial in our experiments. (c) 2D Placements (Corrupt)Figure 6 : Ablation results (mIoU) of placement strategies on segmentation models [111, 18, 86, 117] .difference in LiDAR heights enhances perception, as varied height distributions capture richer features from the sides of objects.3) Uniform distribution. The pyramid placement performs better in segmentation, as a more spread-out and uniform distribution of 4 LiDARs captures a detailed surface.In this section, we further analyze the interplay between our proposed optimization strategy and perception performance to address two questions: 1) How does our optimization strategy inform LiDAR placements to improve robustness against various forms of corruption? 2) How can our optimization strategy enhance perception performance in scenarios with limited placement options?Optimizing Against Corruptions. In the previous subsection, we deploy the placement optimized on the clean point clouds directly on corrupted ones to assess the robustness. To further showcase the capability of our method against corruptions, we compute the M-SOG scores utilizing the P-SOG derived from the corrupted semantic point clouds under adverse conditions and perform the optimization process, as shown in Figure 6a . Quantitatively compared with Ours in Table 2 , the configurations derived from optimization on adverse data outperform the configurations generated solely on clean data. These results indicate that customizable optimization tailored for adverse settings is an effective approach to enable robust 3D perception.Optimizing 2D Placements. Placing LiDARs at different heights on the roof might affect automotive aesthetics and aerodynamics. Therefore, we investigate the effect of our optimization algorithm in the presence of constraints, by limiting the LiDARs to the same height and considering only 2D placements. We fix the height of all LiDARs to find the optimal placement on the horizontal plane of the vehicle roof. We use the Line, Square, and Trapezoid placements in Figure 2 for comparison.Experimental results in Figures 6b and 6c show that our optimized LiDAR placements outperform baseline placements at the same LiDAR height on both clean and corruption settings.Influence of LiDAR Roll Angles. Fine-tuning the orientation angles of the LiDARs on autonomous vehicles is an effective strategy to minimize blind spots and broaden the perception range. For 3D segmentation, Pyramid-roll slightly outperforms Pyramid, whereas Line-roll falls short of Line, which is highly aligned with scores of M-SOG in Table 1 . For 3D detection tasks, the situation is reversed, but still consistent with the M-SOG results. This suggests that adjusting the LiDAR's angle can have varying impacts on the performance of 3D object detection and LiDAR semantic segmentation. 
conclusion: 
 In this work, we presented Place3D, a comprehensive and systematic LiDAR placement evaluation and optimization framework. We introduced the Surrogate Metric of Semantic Occupancy Grids (M-SOG) as a novel metric to assess the impact of various LiDAR placements. Building on this metric, we culminated an optimization approach for LiDAR configuration that significantly enhances LiDAR semantic segmentation and 3D object detection performance. We validate the effectiveness of our optimization approach through a multi-condition multi-LiDAR point cloud dataset and establish a comprehensive benchmark for evaluating both baseline and our optimized LiDAR placements on detection and segmentation in diverse conditions. Our optimized placements demonstrate superior robustness and perception capabilities, outperforming all baseline configurations. By shedding light on refining the robustness of LiDAR placements for both tasks under diverse driving conditions, we anticipate that our work will pave the way for further advancements in this field of research. LiDAR point cloud data is collected every 0.5 simulator seconds utilizing the Semantic LIDAR sensor in CARLA. The attributes of Semantic LIDAR sensor are listed in Table 5 . Notably, we utilize LiDARs with relatively low resolution to increase the challenge in object detection and semantic segmentation, leading to lower scores than those on the nuScenes [7] leaderboard. This is based on the observation that LiDAR point clouds become sparse when detecting distant objects and the use of multiple LiDARs is specifically to enhance the perception of sparse point clouds at long distances in practical applications. Our experiments with low-resolution LiDARs allow for a more evident assessment of the impact of LiDAR placements on the perception of sparse point clouds. 
 
 Appendix: In this section, we provide more details for the class definitions of the LiDAR semantic segmentation and 3D object detection tasks.There are a total of 21 semantic categories in our LiDAR semantic segmentation dataset, which are 1 Building, 2 Fence, 3 Other, 4 Pedestrian, 5 Pole, 6 Road Line, 7 Road, 8 Sidewalk, 9 Vegetation, 10 Vehicle, 11 Wall, 12 Traffic Sign, 13 Ground, 14 Bridge, 15 Rail Track, 16 Guard Rail, 17 Traffic Light, 18 Static, 19 Dynamic, 20 Water, and 21 Terrain. Table 6 presents the detailed definition of each semantic class in Place3D. Additionally, we include a Unlabeled tag to denote elements that have not been categorized. We set 3 types of common objects in the traffic scene for the 3D object detection tasks, i.e.,1 Car, 2 Bicyclist, and 3 Pedestrian, following the same configuration as KITTI [29] .As shown in Figure 9 and Figure 10 , the adverse conditions in the Place3D dataset can be categorized into three distinct groups:-Severe weather conditions, including "fog", which causes back-scattering and attenuation of LiDAR points due to water particles in the air; "snow", where snow particles intersect with laser beams, affecting the beam's reflection and causing potential occlusions; and "wet ground", where laser pulses lose energy upon hitting wet surfaces, leading to attenuated echoes. These are commonly occurring corruptions in real driving conditions.-External disturbances like "motion blur", which results from vehicle movement that causes blurring in the data, especially on bumpy surfaces or during rapid turns.-Internal sensor failure, such as "crosstalk", where the light impulses from multiple sensors interfere with each other, creating noisy points; as well as "incomplete echo", where darkcolored objects result in incomplete LiDAR readings.The Place3D dataset is released under the CC BY-NC-SA 4.0 license 1 .In this section, we present additional implementation details to facilitate the reproduction of the data generation, placement optimization, and performance evaluations in this work.We adopt five commonly employed heuristic LiDAR placements, which have been adopted by several leading autonomous driving companies (see Figure 8 ), as our baseline. We create another two baseline placements by adjusting the rolling angles of LiDAR sensors. We obtain two placements in optimization and two placements in the ablation study. All configurations are presented in Table 7 . In this work, we adopt the public implementations from Robo3D [46] to generate corrupted point clouds. We follow the default configuration of generating nuScenes-C to construct the adverse condition sets in Place3D. Specifically, for "fog" generation, the attenuation coefficient is randomly sampled from [0, 0.005, 0.01, 0.02, 0.03, 0.06] and the back-scattering coefficient is set as 0.05. For "wet ground" generation, the water height is set as 1.0 millimeter. For "snow" generation, the value of snowfall rate parameter is set to 1.0. For "motion blur" generation, the jittering noise level is set as 0.30. For "crosstalk" generation, the disturb noise parameter is set to 0.07. For "incomplete echo" generation, the attenuation parameter is set to 0.85.In this work, we build the LiDAR placement benchmark using the MMDetection3D codebase [20] .Unless otherwise specified, we follow the conventional setups in MMDetection3D when training and evaluating the LiDAR semantic segmentation and 3D object detection models.The detailed training configurations of the four LiDAR semantic segmentation models, i.e., MinkUNet [18] , SPVCNN [86] , PolarNet [111] , and Cylinder3D [117] , are presented in Table 8 . The detailed training configurations of the four 3D object detection models, i.e., PointPillars [51] , CenterPoint [107] , BEVFusion-L [63] , and FSTR [109] , are presented in Table 9 . All LiDAR semantic segmentation models are trained and tested on eight NVIDIA A100 SXM4 80GB GPUs. All 3D object detection models are trained and tested on four NVIDIA RTX 6000 Ada 48GB GPUs. The models are evaluated on the validation sets. We do not include any type of test time augmentation or model ensembling when evaluating the models.In this section, we provide additional quantitative results to better support the findings and conclusions drawn in the main body of this paper.We provide the complete (i.e., the class-wise IoU scores) results for LiDAR semantic segmentation under different LiDAR placement strategies.We showcase the per-class LiDAR semantic segmentation results of MinkUNet [18] , SPVCNN [86] , PolarNet [111] , and Cylinder3D [117] in Table 11 . The performance of these methods is evaluated under different LiDAR placement strategies. Different LiDAR placement strategies demonstrated varying propensities towards particular classes. For instance, Pyramid performs well for building, guard rail, and vegetation compared with other placements, possibly due to an increased vertical field of view that captures these taller or layered structures more effectively. Ours provides the generally best balance of performance across categories, with high scores in building, road, and vehicle suggesting an effective all-around coverage for various object types. The complete benchmark results of LiDAR semantic segmentation under clean and adverse conditions are presented in Table 12 .We showcase the correlation between M-SOG and LiDAR semantic segmentation performance under clean and adverse conditions in Figure 11 and 12.We showcase the complete results of the ablation study for semantic segmentation in Table 14 . we first explore the performance of our optimization algorithm under placement constraints, specifically, standardize the elevation of LiDARs to analyze their optimal positioning on the vehicle's roof's horizontal plane. Our optimized placement 2D plane achieved better performance compared with line, square, and trapezoid configurations. Further, we optimize the placement for adverse conditions. While the observation suggests a trade-off in clean data compared with the Ours, the configuration optimized on corruption data (corruption optimized) achieves a significantly higher performance in adverse conditions than both baselines in Table 12 and Ours, indicating that custom optimization strategies, tailored for challenging conditions, represent an effective methodology to enhance robust perception capabilities in adverse environments.In this section, we present more experimental results for the 3D robustness evaluation of 3D object detection under adverse conditions. The performance of PointPillars [51] , CenterPoint [107] , BEVFusion-L [63] , and FSTR [109] is evaluated under different LiDAR placement strategies in Table 13 . Across all placements and conditions, there is a trend where all methods suffer a drop in performance in adverse conditions compared to clean conditions, highlighting the challenge that adverse conditions pose to 3D object detection. While some placements, like Pyramid and Square, appear to maintain relatively high performance under both clean and adverse conditions, the Ours placement has the highest average mAP under adverse conditions. In addition, certain placements exhibit relative strengths against specific types of corruption, which can inform the development of more resilient object detection systems for autonomous vehicles. For instance, Pyramid and Pyramid-Roll, appear to handle fog better than others, possibly due to a configuration that captures a more diverse set of angles which could mitigate the scattering effect of fog on LiDAR beams. We showcase the correlation between M-SOG and 3D Object Detection performance under clean and adverse conditions in Figure 13 and 14. The effectiveness of sensor placement metrics is determined by the linear correlation between these metrics and actual performance. As shown in Table 10 , Jiang's method [40] is applied to roadside placements, while comparable on-vehicle methods, S-MIG [34] and S-MS [57] , follow the same process for LiDARs. All these methods use 3D bounding boxes as priors to understand the 3D scene distribution. However, this approach leads to significant deviations from the actual physical boundaries of objects and fails to account for occlusion relationships between objects and the environment in LiDAR applications. Consequently, they cannot accurately describe the scene or effectively optimize LiDAR placements. Moreover, these methods are limited to evaluating LiDAR placements for detection tasks since they solely rely on 3D object distribution information.Our method addresses these limitations by introducing semantic occupancy information as a prior for the evaluation metric. This allows for more accurate characterization of boundaries in the 3D scene and effectively addresses occlusion issues between objects and the environment. Additionally, our method leverages semantic distribution information under diverse conditions, thereby enhancing the reliability of perception in adverse weather conditions and during sensor failures.In this section, we provide additional qualitative examples to help visually compare different conditions presented in this work.We showcase the qualitative results of the MinkUNet [18] model using our optimized LiDAR placement. As shown in Figure 15 , the depiction across the first four rows demonstrates a commendable performance of our LiDAR placement strategy during clean conditions. This is evidenced by the predominance of gray in the error maps, which indicates a high rate of correct predictions. Also, the fifth row presents a notable exception, illustrating a failure case. This particular scene exhibits a heightened complexity with a diverse array of objects and potential occlusions that challenge predictive accuracy. The qualitative results underscore the importance of continuous refinement of LiDAR placement to achieve better perception across a comprehensive spectrum of scenarios.In Figure 16 , we showcase the qualitative results of MinkUNet [18] under different adverse conditions utilizing our optimized LiDAR placement. The corruptions from top to bottom rows are "fog", "wet ground", "motion blur", "crosstalk", and "incomplete echo". As can be seen, the LiDAR semantic segmentation model encounters extra difficulties when predicting under such scenarios. Erroneous predictions tend to appear in regions contaminated by different types of noises, such as airborne particles, disturbed laser reflections, and jittering scatters. It becomes apparent that enhancing the model's robustness under these adverse conditions is crucial for the practical usage of driving perception systems. To achieve better robustness, we design suitable LiDAR placements that can mitigate the degradation caused by corruptions. As discussed in Table 12 , Table 13 , and Table 14 , our placements can largely enhance the robustness of various models.Figure 11: The correlation between M-SOG and LiDAR semantic segmentation [18, 86, 111, 117] models performance in the clean condition. Figure 12 : The correlation between M-SOG and LiDAR semantic segmentation [18, 86, 111, 117] models performance in the adverse condition. 6 for color maps. Best viewed in colors and zoomed-in for details.Table 11 : The per-class LiDAR semantic segmentation results of MinkUNet [18] , SPVCNN [86] , PolarNet [111] , and Cylinder3D [117] , under different LiDAR placement strategies. All mIoU (↑) and class-wise IoU (↑) scores are given in percentage (%). EQUATIONwhere u * is the global optima over U.Proof. Based on the sampling over subset S ⊂ U with ℓ 2 -norm density δ, we have:EQUATIONFrom the continuous objective function G : R n → R with Lipschitz constant k G w.r.t. input u ∈ U ⊂ R n under ℓ 2 norm, it holds that:EQUATIONSince the distance between the maximal and minimal of function G over S is upper-bounded by C M , we then have:EQUATIONThen by absolute value inequality with u S ∈ S ⊂ U s.t. ∥u * -u S ∥ 2 ≤ δ and combining all inequalities above, we have:EQUATIONwhich concludes the proof.Corollary 2. When U is a hyper-rectangle with the bounded ℓ 2 norm of domain U i ∈ R at each dimension i, with i = 1, 2, . . . , n, Thm.2 can hold in a more general way by only assuming that the Lipschitz constant k G of the objective function is given, where the following optimality certification regarding x ∈ U holds that:EQUATION)Proof. Similar to the proof of Thm. 2, by the sampling over subset S ⊂ U with ℓ 2 -norm density δ, we have:EQUATIONFrom the continuous objective function G : R n → R with Lipschitz constant k G w.r.t. input u ∈ U ⊂ R n under ℓ 2 norm, it holds that:EQUATIONIn this section, we discuss the limitations and potential negative societal impact of this work.In this work, we utilize CARLA to generate LiDAR point clouds instead of collecting data in the real world. Although the delicacy and realism of simulation technologies are now remarkably close to the real world, there still exists a gap. For example, LiDAR cannot detect transparent objects, and LiDAR generates noise when encountering walls; these characteristics are not well-represented by the simulator. In addition, for each placement, we collected 13,600 frames of data; however, real-world applications would require a significantly larger dataset.Although we selected seven simplified placements as baselines for extensive experimentation, our baseline placements cannot reflect and cover all possible LiDAR arrangements for various car manufacturers. Furthermore, the optimized placements we obtained are only near-optimal for our dataset. Identifying the globally optimal placements requires further analysis. In addition, our dataset employs spinning LiDAR technology. Yet, the latest advancements in autonomous driving have indicated a trend toward adopting semi-solid-state and solid-state LiDAR systems, necessitating further research. To demonstrate the impact of LiDAR placement, we selected LiDAR sensors with relatively low resolution. The outcomes might differ when the total number of LiDAR sensors is not four or when using LiDAR sensors with different channels and scanning frequencies.When generating the SOG (semantic occupancy grids) for computing the M-SOG metric, we use a voting algorithm to determine the semantic label for each voxel. This approach can introduce potential errors in cases where multiple semantic labels within a voxel have a similar number of points. Additionally, to accurately describe the semantic distribution of a scene, smaller voxel sizes are often required.LiDAR systems, by their very nature, are designed to capture detailed information about the environment. This can include not just the shape and location of objects, but potentially also capturing point cloud data of individuals, vehicles, and private property in high resolution. The data captured might be misused if not properly regulated and secured. Further, With autonomous vehicles navigating spaces using LiDAR and other sensors, there could be a shift in how spaces are designed, potentially prioritizing efficiency for autonomous vehicles over human-centered design principles. In addition, over-reliance on LiDAR and perception systems can lead to questions about the trustworthiness and reliability of these systems.We acknowledge the use of the following public resources, during the course of this work:- 38th Conference on Neural Information Processing Systems (NeurIPS 2024).https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode.en.https://github.com/carla-simulator/carla.https://github.com/carla-simulator/scenario_runner.https://github.com/open-mmlab/mmcv.https://github.com/open-mmlab/mmdetection.https://github.com/open-mmlab/mmdetection3d.https://github.com/open-mmlab/mmengine.https://github.com/nutonomy/nuscenes-devkit. 


Decision: The paper presents a metric and optimization strategy to obtain LiDAR placements for autonomous driving that achieve improvements for perception tasks like 3D object detection and semantic segmentation, including under adverse conditions of sensor function and weather. It also develops a dataset and benchmark in an attempt to standardize multi-LiDAR configurations. Reviewers 8eRR, PfQe and iXBb recommend accepting the paper, while DpmT argues for rejection. All the reviewers appreciate the value of the contributions. The main issues raised by DpmT are on improved notations and density of information in the main paper. This was discussed extensively, especially by PfQe and DpmT, where the former champions the paper and finds that the presentation does not occlude the contributions. The AC concurs that the issues raised by DpmT are important, but have been sufficiently addressed by the author rebuttal. The authors are encouraged to include the notational improvements requested by DpmT in the final version of the paper. The paper does solve an under-studied but important problem for autonomous driving, where the proposed dataset and methods will help alleviate significant costs associations with trial-and-error involved in setting up expensive multi-LiDAR systems. In balance, the paper is recommended for acceptance at NeurIPS.

Reviews: review: paper_summary: paper_summary: This paper proposes a framework called Place3D to investigate the placement of multiple LiDAR sensors for semantic segmentation and object detection task under various weather conditions.  Place3D mainly introduce a Surrogate Metric of Semantic Occupancy Grids (M-SOG) to evaluate the perception performance with different sensors placements.  Besides, an optimization approach is applied to refine the LiDAR placement. The method is evaluated on the simulated dataset with CARLA, showing the advantages of the refined placements.
strength_weakness: strength_weakness: - Although the proposed method  investigates LiDAR placement for more tasks under more weathers, the core formulation is similar to that in [34] as cited in the paper.
- Only one type of 16-beam LiDAR sensor is investigated. Actually, sensors with more laser beams (64 or even 128) are becoming low-cost and popular. It is important to evaluate the effect of the placement as the point cloud data get dense.
- During the companions in Tab.2,3, different refined placements are applied for different tasks. How is the performance if applied a Seg-optimized placement to the Det task? It is impractical to change the placement for different task. Is that possible to find an optimal placement for both tasks?
questions: questions: - In the dataset comparison part, it is written that no adverse weather for Nuscenes and Waymo.  They may include rainy weather and even more (e.g., Snow).  Although in the appendix part, please also have a check on that.
- How is the solution spaces for the LiDAR placement? Are the displacements along z-axis or even rotation are considered during the optimization?
- Is there any intuitive interpretation about why refined sensor placement are beneficial to different tasks?
limitations: limitations: The authors discussed the possible limitation in data collection.
paper_summary: paper_summary: The authors present a very interesting study of optimal lidar placement on autonomous vehicles. They introduce a novel measure of the quality of lidar placement called M-SOG, and also discuss an optimization approach to find the best placement. They measure the impact of the placement on the important vision tasks in autonomous vehicles, such as object detection and semantic segmentation, showing that the placement metric is aligned with the improved quality. Lastly, in addition to clear weather they also analyze the impact on data when the conditions are degraded (such as fog or faulty sensors). Moreover, they also provide the data and the code that supports their work.
strength_weakness: strength_weakness: - The paper writing is can be improved significantly, as the method is not actually clearly explained.
- The authors tried to stuff too many things in a single paper (metric and degraded sensors), which degraded focus and impact of the work.
- The experiments can be improved significantly.
(detailed comments on the above weaknesses can be found in the section below)
questions: questions: - Line 107, "voxelized grips" -> "voxels"?
- Line 110, "where each voxel is assumed to be independently and identically distributed", what does this actually mean? The voxels are clearly not independent of each other.
- The notation can be improved, as it is confusing at times and not properly introduced. E.g., line 112, what is T, 1.
- Along similar lines, H is not properly introduced a few lines below.
- Line 143, should P_SOG also have k-index in its name? As it is defined for a particular k, as per the given equation.
- In the paragraph starting with line 150, the authors discuss the metric. However, given that the metric is computed using only a subset of voxels, is there an invalid setup that the metric would not capture properly, such as putting lidars below the car or something similar?
- Fig 2, would be good to zoom in on the roof of the car to make it more visible, as that is the relevant part in this figure.
- The optimization in Section 3.3 is not well explained, and the notation is also not properly introduced.
- E.g., the authors do not properly explain what is the lidar configuration space, or what are the constraints. Some details can be gleaned here and there, but given how critical this part is for their work the authors should do a much better job explaining it.
- Along these lines, line 173 mentions cov-mat of distribution, distribution of what? This is very poorly explained.
- Also, in this paragraph the authors use k-index for samples, but k was already used for classes?
- How significant and how practical is the theoretical analysis? The authors introduce C_M as a gap, but this constant is not discuss anywhere, or how large it is. This makes the theoretical discussion quite weak. More discussion and more clarifications are needed.
- Line 238, Tables 2 and 3 don't actually show this.
- The detectors used in these experiments are not clearly discussed.
- In general, it seems that the authors tried to stuff too many things at the same time in the work and in the experiments. This made them relegate many results and discussions to the appendix, even those that are critical to the work. It seems that it would be much better to focus on one aspect first (e.g., metric + impact on vision results in clear weather) and only then expand from there.
- What are the points in Fig 4, not clear and not discussed.
- The adverse aspect is very important, yet the authors do not spend a lot of time/space on it, and the results are just given. This is unfortunate, and weakens their argument around this part of their work.
- Line 253, this was already mentioned in line 238?
- Line 263, why is the placement good for adverse weather? Any deeper insights? How generalizable are these placements? All these important questions are not discussed at all, and the authors just skip them. This is another symptom of stuffing too many things in one paper and losing focus and depth.
- Line 270, the authors just move all important ablation results to the appendix, which is not acceptable. At this point the authors are just abusing the appendix. Please note that the appendix is not meant to increase the page limit, and critical experiments/results/discussions should NOT be moved there.
- More discussions and insights about the found placement should be added, such as around 2D or roll solution. Any advice for the designers, any interesting findings? This is not provided at all.

###################
FEEDBACK AFTER THE REBUTTAL
###################

I would like to thank the authors for responding to my comments, and also to the other reviewers who helped me further understand the paper. However, after reading through all the other responses/discussion it seems that my main concerns were not addressed. E.g.,while I said that the paper is too stuffed and poorly explained, the authors just said that they will add more explanations and that's that. First, I'm not sure where would this be added since there is no space and the paper is overstretched as is, and second, the required changes are significant and just saying that it will be addressed is not enough, the paper would need to be re-reviewed. Moreover, the authors marked my comments under "Other Minor Modifications", which just shows that they did not appreciate the severity of my comments.
I do like the idea and I do think that the work has future, but with the current execution I just can not see how can I increase my recommendation. The paper needs quite a lot of attention and rewriting, so much so that I am not confident that the authors would not be able to do that with quick updates, especially given their responses that made light of some of my more important comments.
limitations: limitations: The authors do not discuss the limitations in a separate section.
paper_summary: paper_summary: The paper targets an overlooked but important aspect of 3D scene understanding -- the influence of LiDAR sensor placements for the success of 3D perception tasks, such as 3D detection and 3D segmentation.

The authors introduce Place3D, a comprehensive pipeline for optimizing LiDAR placement in autonomous driving scenarios to enhance 3D scene understanding, especially under some adverse conditions. 

Indeed, current single-LiDAR systems and datasets tend to fail to capture the complexities of real-world environments. This work proposes to address this by using the Surrogate Metric of the Semantic Occupancy Grids (M-SOG) to evaluate LiDAR placement quality. M-SOG measures the information gain and scene understanding capability by calculating the entropy of semantic voxel distributions.

The optimization strategy employs the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to iteratively improve LiDAR configurations. The objective function is the M-SOG score, ensuring that the optimized placements are close to the global optimum. Compared to the existing S-MIG, this work makes a significant contribution by extending the binomial occupancy distribution to multinomial occupancy distribution through multiple semantic labels for better scene understanding.

Place3D also introduces a dataset of 364,000 frames collected under various conditions, including severe weather and sensor failures. The optimized placements show superior performance in LiDAR semantic segmentation and 3D object detection tasks compared to existing baselines.
strength_weakness: strength_weakness: 1. In the experiments, although the baseline LiDAR placement methods are based on existing configuration of current autonomous vehicle systems (Fig. 7), it is better to compare the proposed method with some randomly sampled LiDAR placements. This can further justify the effectiveness of CMA-ES compared to a wider range of scenarios.

2. It is not clear how the density of the occupancy grid (Sec. 3.3) will influence the surrogate metric, as the estimation of semantic distribution depends on the number of samples.

3. The experiments are conducted in the simulation. Although CARLA is a rather realistic simulator and has been widely adopted in academia, it is unknown how well the proposed method works in the real-world setting. Perhaps it is not feasible to conduct on-site experiments, but more justifications regarding this aspect are needed for further discussion. 

4. The optimization process and data generation might require substantial computational resources. Either the main body or the appendix omitted such details. It is highly recommended to supplement such details as they are critical for follow-up works.

5. Can the proposed benchmarks and optimization strategies be extended to other 3D scene understanding tasks, such as 3D object tracking, semantic occupancy prediction, etc?
questions: questions: Please refer to the above section for detailed questions.
limitations: limitations: The authors have included some discussions of the potential limitations and societal impact in the appendix.
paper_summary: paper_summary: This paper proposes Place3D, a full-cycle pipeline that encompasses LiDAR placement optimization, data generation, and downstream evaluations. The framework makes three appealing contributions. 1) To identify the most effective configurations for multi-LiDAR systems, this paper introduces a Surrogate Metric of the Semantic Occupancy Grids (M-SOG) to evaluate LiDAR placement quality. 2) Leveraging the M-SOG metric, this paper propose a novel optimization strategy to refine multi-LiDAR placements. 3) Centered around the theme of multi-condition multi LiDAR perception, the authors collect a 364,000-frame dataset from both clean and adverse conditions. Extensive experiments demonstrate that LiDAR placements optimized using palce3D outperform various baselines.
strength_weakness: strength_weakness: - The representation may be a little confused. The M-SOG and the optimization method, under my understand, is task-independent. However, there are two optimized LiDAR placement related to Det and Seg tasks, which is a little confused.
- To demonstrate the effectiveness of M-SOG, it's recommended to draw a scatter plot of M-SOG and model performance in Tab2 and Tab3, which can be more intuitive.
questions: questions: - I'm curious about the influence of the LiDAR numbers. It's better to add a experiment to explore the most cost-effective LiDAR number.
- M-SOG seems to be a global metric. Can It also reflect the scene coverage of a specific local area?
limitations: limitations: Nan

Comments: title: Summary of Author-Reviewer Discussion Session
Looking Forward to Discussion with You
Looking Forward to Discussion with You
Looking Forward to Discussion with You
Looking Forward to Discussion with You
comment: **Dear Reviewers, Area Chairs, and Program Chairs,**

As the Author-Reviewer Discussion session comes to a close, we would like to express our sincere gratitude for your time, effort, and thoughtful feedback throughout this review process.

We are pleased that our rebuttal and revisions have addressed many of the reviewers’ concerns, and we appreciate the positive feedback we received.

---

Our revisions were guided by your insightful suggestions, resulting in several **key improvements**:
- We clarified the core formulation and highlighted the improvements over the work cited in [34], specifically addressing critical limitations in S-MIG, as suggested by Reviewer `8eRR`. We also improved the explanation of our optimization approach using CMA-ES and expanded the theoretical analysis as per the Reviewer `DpmT`’s advice.
- We included a scatter plot illustrating the relationship between M-SOG and model performance, making our results more intuitive and easier to interpret, as suggested by Reviewer `iXBb`. Additionally, we clarified our comparisons between M-SOG and S-MIG and added further insights into LiDAR placement based on Reviewer `DpmT`’s suggestions.
- We included higher-resolution LiDAR experiments, demonstrating the continued importance of sensor placement with denser point clouds, as recommended by Reviewer `8eRR`. We also included experiments on the influence of the number of LiDARs, as suggested by Reviewer `iXBb`.
- We provided intuitive interpretations of why refined sensor placements are beneficial for various tasks, such as enhancing the field of view and capturing richer features, as suggested by Reviewer `8eRR`. We also discussed the feasibility of extending our benchmarks and optimization strategies to other 3D scene understanding tasks, following Reviewer `PfQe`’s advice.
- We addressed specific concerns regarding adverse weather conditions in datasets, clarified notations, and improved the overall clarity of our figures and writing. We also included additional experiments and discussions, such as local area coverage using M-SOG, reflecting the valuable input from all reviewers.

---

We would like to highlight again the **technical contributions** of this work:
- To the best of our knowledge, Place3D is the first attempt to investigate the impact of multi-LiDAR placements for 3D semantic scene understanding across diverse conditions. It addresses the limitations of existing datasets that often lack adverse weather and sensor failure scenarios.
- We introduce the Surrogate Metric of Semantic Occupancy Grids (M-SOG), an innovative metric designed to evaluate the quality of LiDAR placements effectively for both detection and segmentation tasks. This metric overcomes the limitations of previous methods by incorporating semantic information and handling occlusions.
- Our novel optimization strategy, utilizing the M-SOG metric, refines LiDAR placements to enhance performance in both LiDAR semantic segmentation and 3D object detection. This approach demonstrates superior robustness and accuracy, outperforming various baselines by approximately 9%.
- We contribute a large-scale, 364,000-frame dataset that includes both clean and adverse conditions, along with a comprehensive benchmark for evaluating LiDAR-based 3D scene understanding. This dataset is crucial for testing the robustness of 3D perception systems under realistic conditions.

---

We hope these revisions have strengthened our manuscript and fully addressed your feedback. We are encouraged by the **positive recognition** of our work’s potential impact, as highlighted by Reviewer `PfQe`’s comments, who has upgraded the rating from *"6: Weak Accept"* to *"7: Accept"*.

---

We sincerely thank you for your constructive suggestions and for helping us improve the quality of this work.

Best regards,

The Authors of Submission 1401
Thanks to the authors for the comprehensive responses to my review comments.

The major concerns from my comments have been resolved. 

I have also gone through the author’s responses to other three reviewers’ comments. 

I am confident that this work is of a good value in the area of research and could open up future research opportunities.

Therefore, I decide to change the rating from “Weak Accept” to “Accept”.
**Dear Reviewer `iXBb`,**

We sincerely appreciate your time and effort in reviewing our manuscript and providing valuable feedback.

---

In response to your insightful comments, we have made the following revisions to our manuscript:

- We have clarified the task-independent nature of M-SOG and the optimization method and explained the variations in optimized placements for different tasks.
- We have added experiments to explore the influence of the number of LiDARs on performance, showing the most cost-effective LiDAR numbers.
- We have included discussions on the local area coverage using M-SOG, adapting the metric to reflect specific regions.
- We have added scatter plots of the M-SOG and model performance, which can be found in the attached single-page Rebuttal PDF file.

---

We hope these revisions adequately address your concerns.

We look forward to actively participating in the **Author-Reviewer Discussion** session and welcome any additional feedback you might have on the manuscript or the changes we have made.

---

Once again, we sincerely thank you for your contributions to improving the quality of this work.

Best regards,

The Authors of Submission 1401
**Dear Reviewer `PfQe`,**

We sincerely appreciate your time and effort in reviewing our manuscript and providing valuable feedback.

---

In response to your insightful comments, we have made the following revisions to our manuscript:

- We have included additional experiments comparing the proposed method with randomly sampled LiDAR placements to justify the effectiveness of our method.
- We have included details on the computational resources required for the optimization process and data generation.
- We have discussed the feasibility of extending our benchmarks and optimization strategies to other 3D scene understanding tasks.
- We have supplemented details on the influence of the density of the occupancy grid on the surrogate metric.

---

We hope these revisions adequately address your concerns.

We look forward to actively participating in the **Author-Reviewer Discussion** session and welcome any additional feedback you might have on the manuscript or the changes we have made.

---

Once again, we sincerely thank you for your contributions to improving the quality of this work.

Best regards,

The Authors of Submission 1401
**Dear Reviewer `DpmT`,**

We sincerely appreciate your time and effort in reviewing our manuscript and providing valuable feedback.

---

In response to your insightful comments, we have made the following revisions to our manuscript:

- We have improved the explanation of the optimization approach using the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) and added more details to the theoretical analysis.
- we have clarified our experiments for comparison between our M-SOG and existing placement method S-MIG in Fig 4 and provided several findings and insight into LiDAR placement in our experiments.
- We have improved the clarity of our figures, such as zooming in on the relevant parts in Fig 2. To better explain Tabs 2 and 3, we have added a scatter plot of M-SOG and model performance, as shown in the attached single-page Rebuttal PDF.
- We have clarified notations and improved the overall writing to make the method more understandable. For typos, We corrected "voxelized grid" to "voxels". For notations, we properly introduced notations $T$ and $H$, included the $k$ index in $P_{SOG}$, and revised the $k$ index in optimization to avoid overlap. We also did a thorough check to ensure all notations were properly introduced and clearly explained.

---

Additionally, we provide a list of **Summary of Notations** as follows:

|Notation|Explanation|
|-|-|
|$L$|Length of the Region of Interest (ROI)
|$W$|Width of the Region of Interest (ROI)
|$H$|Height of the Region of Interest (ROI)
|$\delta_L$|Resolution of voxelization in the length dimension
|$\delta_W$|Resolution of voxelization in the width dimension
|$\delta_H$|Resolution of voxelization in the height dimension
|$S$|Set of voxels in the Region of Interest (ROI)
|$N$|Total number of voxelized grids
|$K$|Total number of semantic labels
|$v_i$|Voxel $i$
|$p(v_i)$|Probability of voxel $v_i$ being occupied
|$y_t$|Frame $t$
|$T$|Total number of frames
|$H(v_i)$|Entropy of voxel $v_i$
|$H(v_{i} &#x7c; L)$|Conditional entropy of voxel $v_i$ given LiDAR placement $L$
|$H_{POG}$|Total entropy of POG (Probabilistic Occupancy Grids)
|$p_{POG}$|Joint probability of all non-zero voxels in the ROI
|$p_{POG &#x7c; L}$|Conditional probability of POG given LiDAR placement $L$
|$H_{POG &#x7c; L}$|Conditional entropy of POG given LiDAR placement $L$
|$\Delta H$|Information gain of 3D scene understanding
|$S_{SOG}$|Semantic occupancy grid
|$s_{y_k}^{(t)}$|The set of voxels occupied by semantic label $y_k$ at frame $t$
|$p(v_i = y_k)$|Probability of voxel $v_i$ being occupied by semantic label $y_k$
|$\mathcal{P}_{SOG}$|Conditional joint distribution of SOG given LiDAR placement $L$
|$\mathcal{H}_{SOG}$|Entropy of SOG
|$L_j$|LiDAR configuration $j$
|$\mathcal{H}^{L=L_j}_{SOG}$|Conditional entropy distribution of P-SOG over the voxel set $S &#x7c; L_j$
|$\mathbf{M}_{SOG}(L_j)$|Normalized surrogate metric of SOG
|$\mathbb{E}$|Expectation operator
|$\mathbf{u}_j$|LiDAR configuration $j$ in objective function $F(\mathbf{u}_j)$
|$F(\mathbf{u}_j)$|Objective function for LiDAR configuration $j$
|$P(\mathbf{u})$|Physical constraint for LiDAR configuration
|$G(\mathbf{u})$|Function representing the surrogate metric
|$U$|Set of all potential LiDAR configurations
|$\mathbf{u}^*$|Optimal LiDAR configuration
|$\mathbf{m}^{(k)}$|Mean vector of the distribution at iteration $k$
|$\sigma^{(k)}$|Step size of the distribution at iteration $k$
|$\mathbf{C}^{(k)}$|Covariance matrix of the distribution at iteration $k$
|$\mathcal{N}(\mathbf{m}^{(k)}, (\sigma^{(k)})^2\mathbf{C}^{(k)})$|Normal distribution for sampling ($k$ will be updated to $g$ to avoid overlap with semantic class)
|$\delta$|Density for discretizing the configuration space $U$
|$\mathbf{u}_i^{(k)}$|Sampled candidate configuration at iteration $k$
|$M_k$|Number of best solutions used to update $m^{(k+1)}$
|$w_i$|Weights based on solution fitness
|$\mathbf{p}_\mathbf{C}^{(k+1)}$|Evolution path accumulating information about the direction of successful steps
|$c_\mathbf{C}$|Learning rate for updating the covariance matrix
|$p_{\sigma}$|Evolution path for step size adaptation
|$c_{\sigma}$|Learning rate for updating the evolution path $p_{\sigma}$
|$d_{\sigma}$|Normalization factor for step size adaptation
|$E\|N(0, I)\|$|Expected length of a standard normally distributed random vector
|$\| \cdot \|$|Euclidean norm

---

We hope these revisions adequately address your concerns.

We look forward to actively participating in the **Author-Reviewer Discussion** session and welcome any additional feedback you might have on the manuscript or the changes we have made.

---

Once again, we sincerely thank you for your contributions to improving the quality of this work.

Best regards,

The Authors of Submission 1401
**Dear Reviewer `8eRR`,**

We sincerely appreciate your time and effort in reviewing our manuscript and providing valuable feedback.

---

In response to your insightful comments, we have made the following revisions to our manuscript:

- We have clarified the core formulation and highlighted the improvements over the work cited in [34], specifically addressing critical limitations in S-MIG.
- We have expanded our experiments to include higher-resolution LiDARs, demonstrating that placement remains important even with denser point clouds.
- We have provided intuitive explanations of how refined sensor placements enhance performance in various tasks, such as improving the field of view and capturing more detailed features.
- We have verified the presence of adverse weather conditions in the Waymo and nuScenes datasets and detailed the key differences between our dataset and others in terms of adverse weather scenarios.

---

We hope these revisions adequately address your concerns.

We look forward to actively participating in the **Author-Reviewer Discussion** session and welcome any additional feedback you might have on the manuscript or the changes we have made.

---

Once again, we sincerely thank you for your contributions to improving the quality of this work.

Best regards,

The Authors of Submission 1401