Collision Avoidance Metric for 3D Camera Evaluation

Authors: vage  taamazyan
alberto  dall'olio
agastya  kalra
Venue: thecvf_com_CVPR_2024_Workshop_VLADR
Published: 2024-03-25 16:47:16

Abstract: 3D cameras have emerged as a critical source of information for applications in robotics and autonomous driving. These cameras provide robots with the ability to capture and utilize point clouds, enabling them to navigate their surroundings and avoid collisions with other objects. However, current standard camera evaluation metrics often fail to consider the specific application context. These metrics typically focus on measures like Chamfer distance (CD) or Earth Mover’s Distance (EMD), which may not directly translate to performance in real-world scenarios. To address this limitation, we propose a novel metric for point cloud evaluation, specifically designed to assess the suitability of 3D cameras for the critical task of collision avoidance. This metric incorporates application-specific considerations and provides a more accurate measure of a camera's effectiveness in ensuring safe robot navigation.

Main paper: 
 introduction: 
 Almost every existing robotics or autonomous driving system relies on a 3D scene representation for functions such as object detection, path planning, and overall scene understanding [11, 14, 18, 23, 24, 30] . This representation typically takes the form of a point cloud, acquired using sensors like LiDAR, time-of-flight, structured light, or stereo cameras. The 3D scene representation is crucial for the robot to navigate safely and perform its tasks without colliding with objects [8] . Therefore, the quality of the point cloud is essential, and evaluating it becomes paramount.However, current point cloud evaluation methods often assess the point cloud itself, neglecting its downstream application. Metrics like Chamfer Distance (CD) [5, 7] , Hausdorff Distance [4, 6] , Earth Mover's Distance (EMD) [21] , Completeness and Accuracy [10] are commonly employed, but they solely evaluate point cloud properties and don't directly translate to application performance. Furthermore, comparing sensors solely based on these metrics can be misleading, as one sensor might excel in one metric while another dominates in another metric. We argue that existing metrics fail to directly capture the core purpose of the point cloud: enabling safe and successful operation.To address this gap, we propose the Collision Avoidance metric. This new metric offers several key advantages:1. Addresses Downstream Performance: This metric directly assesses the suitability of a point cloud from a specific 3D camera for a particular robotics or autonomous driving application. It focuses on downstream performance, meaning it goes beyond the point cloud itself and evaluates its effectiveness in real-world tasks. See Sec. 3 for more details. 2. Supports Threshold Identification: The metric can be used to identify various tolerances for collision detection that can then be programmed into the robot. This enables the robot to adjust its sensitivity based on the specific environment and sensor capabilities, leading to better adaptation and performance. See Sec. 4.5 for more details. Our intuition behind the metric is straightforward: ide-ally, we would like to evaluate a point cloud's effectiveness by testing a robot in various real-world scenarios. This would involve capturing a point cloud of an arbitrary scene, placing a real robot in a large number of random locations, and checking for collisions. However, this approach is impractical and potentially damaging due to its cost, complexity, and potential for causing damage during collisions. As a more feasible alternative, a simulation-based approach could be used. If both captured and ground truth point clouds are available, we can simulate various robot movements and compare the predicted and actual collisions. While this method is less ideal as it relies on ground truth accuracy, it offers a more realistic evaluation compared to solely evaluating point cloud properties. However, implementing such simulations can become complex.Therefore, we introduce an additional simplification: we assume a rectangular cuboid shaped gripper of size L × M × N with linear movement. We move the gripper along a direction x until a collision is detected with the predicted point cloud and the ground truth point cloud. This allows us to compare the distance to collision between both scenarios. We repeat this process for various directions and convert these comparisons into False Positive and False Negative Collision rates for predicted collisions. One of the benefits of this approach is its ability to inform robot tolerance requirements. For instance, we can increase the cube size until false negative collisions are eliminated, providing insights into the necessary safety margins for the robot.We demonstrate, using captured scenes with various 3D sensors, that the proposed Collision Avoidance metric effectively ranks 3D sensors for specific applications based on their suitability for those tasks. In summary, our key contributions are: 1. We propose a novel Collision Avoidance metric for evaluating point clouds in robotic and autonomous driving applications. 2. We compare our metric to existing methods and demonstrate, using a variety of captured scenes with diverse materials and lighting conditions, how effectively the metric ranks different 3D sensors based on their performance.While evaluating a sensor requires capturing raw 3D data using that sensor, we do not propose a specific dataset. We believe the ease of replicating the method with any point cloud makes it widely applicable and readily adoptable. 
related work: 
 The evaluation of point clouds predominantly employs metrics such as the Chamfer Distance [5, 7] and its variations, including the Density-aware Chamfer Distance [29] , the Hausdorff Distance [4, 6] , and the Earth Mover's Distance (EMD) [21] . These metrics have been widely used in a significant body of research related to point cloud evaluation [2, 7, 16] , with their differentiability making them suitable for direct use as loss functions during the training process [7, 20] .In robotics, especially in applications involving LiDAR technology, metrics based on the Root Mean Squared Error (RMSE), such as the point-to-point error, are frequently utilized for evaluating point clouds [13, 15, 17, 25] .Other metrics, albeit less commonly employed, include the Structural Similarity Index Metric (SSIM) [3] , which assesses point clouds based on geometric or color features, and is less often used for direct comparisons of point clouds captured by 3D sensors. Another example is DPDist, which compares point clouds by measuring the distance between the surfaces from which they were sampled [26] . The Sliced Wasserstein distance [19] is mentioned as well, noted for properties similar to those of the EMD.The following sections provide a more detailed examination of the commonly utilized metrics.The Chamfer Distance (sometimes called the Chamfer Measure) has several formulations, but they all share a core principle. For each point in a point cloud P 1 of size N, the distance to the closest point in another point cloud P 2 of size M is found. These distances are then summed. This process is repeated for each point in P 2 with respect to the points in P 1 .Two common distance measures are employed: the L 1 norm and the L 2 norm (Euclidean distance) [7] . Additionally, while sometimes the sum is divided by the number of points in the point cloud [28] , this normalization step does not affect the fundamental concept of the approach.CD(P 1 , P 2 ) = 1 N x∈P1 min y∈P2 x-y 2 2 + 1 M y∈P2 min x∈P1 x-y 2 2 (1)The Hausdorff Distance between two point clouds, denoted as P 1 and P 2 , is typically defined as the greatest of all the distances from a point in one set to the closest point in the other set. Specifically, if we define the minimum distance from a point x in P 2 to the set P 1 as:EQUATIONthen the one-sided Hausdorff Distance from P 1 to P 2 is given by:EQUATIONwhere HD os (P 1 , P 2 ) denotes the one-sided Hausdorff Distance from P 1 to P 2 .The full Hausdorff Distance between P 1 and P 2 is then defined as the maximum of the two one-sided Hausdorff Distances [4] or their sum [28] . This can be represented as:HD(P 1 , P 2 ) = max x∈P1 M D(x, P 2 ) + max y∈P2 M D(y, P 1 ) (4)This measure effectively captures the notion of the greatest distance between two point clouds.Assume that point clouds P 1 and P 2 are of equal size. The Earth Mover's Distance (EMD) between these point clouds can be defined as:EQUATIONwhere φ : P 1 → P 2 represents a bijection between points in P 1 and points in P 2 [7] .In more general terms, when the point clouds P 1 and P 2 do not necessarily have the same size, the correspondence between points in these clouds can be established through a mapping matrix, Π(P 1 , P 2 ), an M × N matrix. In this matrix, the sum of the columns and rows equals one, and each element, Π i,j , ranging from 0 to 1, specifies the degree to which points x i from P 1 and y j from P 2 correspond to each other [28] .Evaluating point clouds in benchmarks for Multiview Stereo often involves metrics such as accuracy and completeness, commonly referred to as precision and recall, along with their respective F-scores [10, 22, 27] . Among the metrics described, this set uniquely incorporates an explicit distinction between Ground Truth and Reconstructed point clouds. Following the definition by Knapitsch et al. [10] , these metrics are defined as follows:Let P GT represent the ground truth point cloud containing M points, and P Q denote the captured or reconstructed point cloud with N points, which is subject to evaluation. For any point x ∈ P Q , the distance to P GT is defined as:EQUATIONSubsequently, the precision (or accuracy) metric for a given threshold d is defined as:EQUATIONwhere [•] represents the Iverson bracket, situating P (d) within the range of 0 to 1. Conversely, the recall (or completeness) metric is defined by reversing the roles of P GT and P Q . For any point y ∈ P GT , the distance to P Q is defined as:EQUATIONThus, the recall for the given threshold d is:EQUATIONTypically, precision and recall are combined to compute the F-score via the harmonic mean:EQUATIONand the robot arm. 3. Apply a coat of white matte paint, specifically formulated for 3D object scanning, to the scene.4. Utilize the robot arm to rotate the scene and capture a point cloud from each position using Photoneo. 5. Merge the captured point clouds into a single, complete, and accurate point cloud knowing the robot intermediate joint configurations. This outlined procedure guarantees an accurate and complete point cloud representation of the scene, regardless of the material properties or lighting conditions. To capture a set of test scenes, we allowed the spray paint to dry completely before replicating the robot arm positions under both room light and sunlight. Since the robot is repeating the pose with sub-millimeter accuracy, it doesn't create a significant additional error. During this process, we captured 3D data with each sensor in the scene. To ensure well-aligned captured point clouds with the rotated GT point cloud, we positioned the stereo cameras and Photoneo XL as close as possible to each other and performed pre-calibration among them.The evaluation produces 5 different point clouds: • Photoneo point cloud obtained directly from the 3D sensor; • Point cloud obtained by the SGM algorithm using the passive stereo setup with RGB camera stereo pair; • Point cloud obtained by the SGM algorithm using the active stereo setup with IR camera stereo pair; • Point cloud obtained by the CREStereo algorithm using the passive stereo setup with RGB camera stereo pair; • Point cloud obtained by the CREStereo algorithm using the active stereo setup with IR camera stereo pair; The metrics outlined in Sec. 2, alongside the novel Collision Avoidance metric, are computed based on the Ground Truth point cloud generated as described above. The comparison between the Collision Avoidance metric and existing ones is presented in Tab. 2. Furthermore, Tab. 1 illustrates the performance of Collision Avoidance metric under various lighting conditions, demonstrating its effectiveness in identifying challenges faced by 3D structured light scanners in intense lighting scenarios. 
collision avoidance metric description: 
 This section describes the Collision Avoidance metric for a captured point cloud, also referred to as the "query" point cloud, denoted as P C Q . The metric aims to compare the robot's detected collisions with objects in the scene to what would occur in the real world, using the collisions with ground truth (GT) point cloud as a proxy for real-world collisions. This metric focuses on ensuring the safe and successful operation of the robot in any given scene, regardless of material types or lighting conditions. The metric identifies two types of errors: 1. False Positive Collisions (FPC): Collisions predicted by the query point cloud that wouldn't occur in reality (also called "ghost collisions"). 2. False Negative Collisions (FNC): Collisions missed by the query point cloud that would occur in reality ("missed collisions"). These errors are aggregated into FPC rate R F P C and FNC rate R F N C , as well as Collision F-score F C. The metrics are described in detail in Sec. 3.2.The metric requires the following data: 1. P C GT : Ground truth point cloud of the scene. 2. P C Q : Query point cloud of the scene captured by a sensor or group of sensors. 3. T Z : Gripper tolerances in Z direction. 4 . L × M × N : Gripper size (length, width, height). 5. G step : Gripper step in XY plane. This is also a proxy to XY tolerance, see details in Sec. 3.2. 6. N GT and N Q : Outlier thresholds (number of points exceeding the threshold) for the ground truth and query point clouds, respectively (consistent across all scenes for each sensor). 7. Ds: Set of directions for gripper movement. Each direction is a 3D vector.The evaluation process follows these steps: 1. Load point cloud: Load the relevant point cloud (either P C GT or P C Q ) and its corresponding outlier threshold. 2. Iterate over directions: Iterate through gripper movement directions Ds. Each direction is a 3D vector, defining the direction along which the gripper movement is simulated (see Fig. 1 and Fig. 2 ). 3. Sample initial positions: For each direction, sample a set of initial gripper positions in the plane perpendicular to the direction vector, using the G step parameter. This creates a set of final paths for the gripper descent. 4. Simulate gripper descent: For each path, simulate the gripper's descent and find the moment of collision, defined as the first instance where the number of points inside the gripper is greater than the threshold value (N GT or N Q ). 5. Label paths: Repeat steps 1-4 for both P C GT and P C Q . Each path of the ground truth is compared with corresponding path of the query and its 4 neighbor paths is XY plane, and the best output is selected. This step defines the T XY as the G step defined above. Then every path is labeled as follows: (a) Aligned: P C Q point cloud is on par with P C GT point cloud for collision avoidance for that path. (b) FPC: P C Q predicts a collision that wouldn't occur based on P C GT within the T Z from the collision point (ghost collision). (c) FNC: P C G T predicts a collision that wouldn't occur based on P C Q within the T Z from the collision point (missed collision). 6. Calculate rates: Calculate the FPC and FNC rates as the percentage of number of FPC and FNC collisions relative to the total number of paths across all directions.EQUATIONEQUATIONwhere N F N C is the number of paths labeled as False Negative Collisions, N F P C is the number of paths labeled as False Positive Collisions, N T otal is the total number of paths. The Collision F-score is then defined as:EQUATION7. Tolerance analysis: (Optional) Repeat steps 1-6 for different T Z tolerances to determine the optimal tolerance for a specific application.This section compares the proposed metric to the existing metrics discussed in Sec. 2, highlighting its key conceptual differences:1. Target Application: The proposed metric is specifically tailored for industrial point cloud evaluations: it penalizes omissions that could impact real-world robotic applications. For example, if a thin structure like a screwdriver is omitted from the point cloud, this wouldn't significantly change any of the metrics described in Sec. 2, but would be penalized more strongly by the Collision Avoidance metric. We believe this is important because such an omitted screwdriver could cause a collision with the robot and break the gripper. See Fig. 3 for an example of how the Collision Avoidance metric reacts to small objects missed in the point clouds (compared to other metrics). 2. Density Tolerance: Unlike existing metrics, this metric doesn't penalize a less dense query point cloud (P Q ) compared to the ground truth (P GT ) as long as the captured density fulfills the specific gripper/robot endeffector specifications. This focuses on functionality rather than absolute density. 3. Small Hole Tolerance: Similar to point 1, the metric doesn't penalize small holes in the query point cloud, as long as they are smaller than the robot end-effector. This prevents penalizing the metric for irrelevant details smaller than the gripper/robot's operational size. 4. Directional Tolerance: The metric allows independent tolerances for XY and Z dimensions based on a given direction vector. This acknowledges the inherent differences in resolution and importance depending on the application and robot movement. This combined approach provides a more comprehensive evaluation. 6. Directionality: Unlike completeness and accuracy metrics from Sec. 2.4, the Collision Avoidance metric explicitly considers directions, further enhancing its relevance for downstream applications. For example, it may be more important to capture a depth map and ensure safe robot movements along a given direction, which would be less sensitive to certain types of holes. We show the metric dependence on directions in Sec. 4.5. 
evaluation: 
 This section demonstrates how our proposed metric can be used to evaluate a structured light sensor (Photoneo XL [1] ) against active and passive stereo cameras. When choosing 3D cameras for robotics, there are often trade-offs to consider. Structured light sensors typically offer higher accuracy but lower completeness due to occlusions caused by large baselines and challenging surfaces. Passive stereo cameras can handle these issues better but may have lower accuracy. We aim to demonstrate how the Collision Avoidance metric can be used to compare these cameras effectively, considering the specific requirements of the downstream robotic application. We utilize two stereo depth reconstruction algorithms, CREStereo [12] and SGM [9] , to generate point clouds from the stereo camera data. Details on the test scenes capture are provided in Sec. 4.1. Our goal is to rank the 3D cameras based on a set of defined tolerances, as discussed in Sec. 4.2. We also separately test small (∼10 cm) and large (∼100 cm) baselines for stereo cameras, as shown in Sec. 4.4. Additionally, in Sec. 4.5, we further explore how the chosen tolerances and other input parameters (introduced in Sec. 3.1) can influence the ranking.We captured a set of scenes using a stereo camera in both active and passive modes, alongside a Photoneo XL structured light 3D sensor. The active stereo camera projects an infrared (IR) dot pattern onto the scene, providing additional texture for the stereo matching algorithm. This study utilizes CREStereo and SGM as the stereo matching algorithms. The scenes encompass objects with diverse sizes, shapes, and material properties, captured under both room light and sunlight conditions.To obtain a ground truth (P GT ) point cloud of a scene, we implemented the following procedure: 1. Mount the scene onto a robot arm.We set the following values to the parameters from Sec. 3.1:T Z = 10mm, L × M × N = 10mm × 10mm × 10mm, G step = 5mm, N GT = 15, N Q = 5, Ds = [0, 0, 1]. Tab. 1shows the aggregated R F P C , R F N C and F C for the tested sensors under both room light and sunlight conditions. We evaluate the performance considering only a single direction, from the top down (Z-axis). Sec. 4.5 analyzes how the metrics change when additional directions are included. For this study, we focus on evaluating the point cloud's suitability for vertical robot movements. Additionally, we ensure that the evaluated cameras are positioned close together, with their direction vectors aligned with the Z-axis. Fig. 4 demonstrates how all the path collision points including the FP and FN Collisions are spanned over the point cloud for a given gripper direction.Tab. 2 aggregates the metrics for all the point clouds across all the scenes. Several important conclusions can be drawn from the table: 1. Chamfer and Hausdorff metrics are not well-suited for evaluating how 3D sensors compare in terms of collision avoidance. SGM Active has a lower Chamfer distance than Photoneo and CRES Passive, however the FC metric shows that SGM Active would basically be unusable for any robotic application with 41.43% R F P C . Similarly, Hausdorff distance is very similar for CRES Active and Photoneo, while the R F N C is much higher for Photoneo, making its use in industrial applications for Collision Avoidance much riskier. 2. Completeness, Accuracy, and F-Score provide results more aligned with the Collision Avoidance metric, as expected due to their conceptual similarity. However, the Collision Avoidance metric provides a clearer inter- and more actionable insights due to its focus on real-world applications. Additionally, it exhibits greater sensitivity to even minor yet critical discrepancies for the robot, as demonstrated in Fig. 3. 3. We argue that most robotic applications prioritize a low R F N C while maintaining a reasonable R F P C . A low R F N C ensures the robot avoids collisions, while a moderate R F P C maintains operational efficiency. Therefore, the Collision Avoidance metric provides the most valuable data for ranking sensors and algorithms for specific downstream robotic applications.The metric directly identifies missed and ghost collision points, allowing their visualization on the query point cloud (see Figs. 3 to 5 for an example). Each red dot represents a FNC, indicating a collision missed by the query point cloud 3D sensor despite occurring in reality. Conversely, each blue dot represents a FPC, signifying a collision predicted by the system but absent in the real world. Fig. 5 demonstrates how Photoneo struggles to capture some shiny pipe sections in sunlight conditions, leading to a higher number of FNCs. These False Negative Collisions occur on the borders of the pipe, in regions not properly reconstructed by the sensor.R F P C room light [%]  R F N C room light [%]  F C room light [%]  R F P C sunlight [%]  R F N C sunlight [%]  F C sunlight [%]  Photoneo XL 0.This subsection delves deeper into understanding Collision Avoidance metric by examining the effects of varying stereo baselines on Passive Stereo SGM performance. Generally, increasing the camera baseline improves the accuracy of the generated point cloud. However, this comes at the cost of reduced completeness due to potential occlusions caused by viewpoint differences. Conversely, a smaller baseline results in a more complete point cloud, but with increased noise.This trade-off between completeness and accuracy is reflected in the Collision Avoidance metric presented in Tab. 3. As the baseline increases, R F P C decreases by approximately 40%, indicating reduced noise. However, R F N C roughly doubles, highlighting the completeness issues in the generated point cloud. Interestingly, the Chamfer and Hausdorff distances show less sensitivity to baseline changes, while the Accuracy and Completeness metrics exhibit a more noticeable difference (approximately 10%) with the larger baseline approach even though the F-Score results remains the same.The proposed metric can be sensitive to the input parameters. This subsection analyzes how 3D sensor rankings would change depending on these parameters. We focus on two parameters most relevant for real-world robotic applications:1. Gripper Tolerance Ablation (T Z ): This ablation investigates how Collision Avoidance metric changes according to the required tolerance. Using such analysis, user can evaluate a point cloud's suitability for collision avoidance in a specific application. For example, if an application requires a certain R F N C , the user can define a gripper tolerance that guarantees the expected performance. 2. Gripper Direction Ablation (Ds): The results in Sec. 4.2 consider only one possible gripper descent direction. This ablation shows how the metrics change if multiple directions are used.We conducted the initial ablation by computing the metrics with eight different tolerance values for T Z ranging from 2.5 to 20.0 millimeters. The results of this ablation on T Z are shown in Fig. 6 and Fig. 7 . As the tolerance increases, the metrics become more permissive. For example, the R F P C for Active SGM approaches and ends up being better than Photoneo by increasing the threshold. This observation indicates that as the tolerance becomes more permissive, the noise level of the point cloud becomes less critical. Fig. 6 illustrates that this parameter has smaller impact on R F N C in the test scenes, suggesting that adjusting gripper tolerance may not necessarily improve the outcome if the query point cloud fails to capture certain areas of the scene. However, it is important to note that this analysis is scene-dependent and the order may change for other scenes and 3D cameras.To explore the impact of gripper directions, we computed the collision metric using 1, 4, and 7 distinct directions for the simulated direction vectors on Photoneo point clouds. This analysis expands the gripper's exploration beyond simply facing perpendicular to the point cloud plane 3D sensor (assuming a top-down view). For the 4-direction case, we introduce 3 additional directions tilted 30 degrees from perpendicular. Similarly, for 7 directions, we introduce 3 additional directions tilted 45 degrees.RF P C [%]  RF N C [%]  F C [%]  Chamfer [mm]  Hausdorff [mm]  Completeness [%]  Accuracy [%]  F-As discussed in Sec. 4.2, increasing the number of gripper directions has minimal impact on the metrics with a small stereo baseline and when the descent direction aligns with the cameras' Z axis. However, this effect might not hold true for larger baselines, as seen with the Photoneo XL camera. Fig. 8 demonstrates a significant increase (over 50%) in the R F N C for Photoneo XL. This can be attributed to potential occlusions in the top-down view of the point cloud, leading to missed regions. By incorporating additional directions, we detect these occluded areas. 
conclusions: 
 This work introduces the Collision Avoidance metric, a novel approach to point cloud evaluation that prioritizes both downstream performance and interpretability. We demonstrate its intuitiveness and effectiveness in evaluating point clouds from active and passive 3D sensors, particularly for robotic applications. Comparisons to other point cloud evaluation metrics show that the Collision Avoidance metric can lead to better interpretability and different 3D camera rankings that may be more relevant to downstream applications focused on collision avoidance. Extensive ablation studies validate the impact of the metric's parameters. We believe the Collision Avoidance metric will empower researchers and engineers to more effectively assess the suitability of point clouds for tasks requiring robust collision avoidance, as well as help them find optimal parameters for their systems. 
 
 Appendix:  



Reviews: review: title: title: Review
main_review: main_review: Overall Evaluation:

The paper proposes a novel and well-designed metric for evaluating 3D cameras specifically for their suitability in robotic and autonomous driving applications focused on collision avoidance. The metric goes beyond just evaluating the point cloud properties and directly assesses the camera's effectiveness in enabling safe robot navigation. This is an important contribution, as existing evaluation metrics often fail to capture the actual downstream performance required for real-world applications.

Strengths:

- The Collision Avoidance metric provides a more application-oriented evaluation that focuses on the core purpose of 3D cameras - enabling safe robot operation. This is a significant improvement over generic point cloud metrics.

- The metric can be used to identify optimal tolerances for collision detection, allowing robots to better adapt to the sensor capabilities and environmental conditions.

- The interpretability of the metric, with clear identification of false positive and false negative collision points, is very useful for understanding and improving sensor performance.

- The thorough evaluation against various 3D sensors and stereo camera baselines demonstrates the effectiveness and versatility of the proposed metric.

Weaknesses:

- While the paper discusses the intuition behind the metric, more formal mathematical formulation and justification could strengthen the presentation.

- The paper could benefit from a more detailed analysis of the impact of the input parameters (e.g., gripper size, tolerance values) on the final rankings, beyond the ablation study provided.

- Comparison to other application-specific metrics, if any exist in the literature, could provide additional context and insights.

Recommendation:

Overall, this is a well-executed and impactful work that addresses an important gap in 3D camera evaluation. I recommend accepting this paper for publication, as it provides a valuable new tool for the robotics and autonomous driving communities.