Revisiting 3D Object Detection From an Egocentric Perspective

1 Introduction . 3D object detection is a key problem in robotics , including popular applications such as autonomous driving . Common evaluation metrics for this problem , e.g . mean Average Precision ( mAP ) based on box Intersection-over-Union ( IoU ) , follow an object-centric approach , where errors on different objects are computed and aggregated without taking their spatiotemporal relationships with the egoagent into account . While these metrics provide a good proxy for downstream performance in general scene understanding applications , they have limitations for egocentric applications , e.g . autonomous driving , where detections are used to assist navigation of the ego-agent . In these applications , detecting potential collisions on the ego-agent ’ s trajectory is critical . Accordingly , evaluation metrics should focus more on the objects closer to the planned trajectory and to the parts/boundaries of those objects that are closer to the trajectory . Recent works have introduced a few modifications to evaluation protocols to address these issues , e.g. , breaking down the metrics into different distance buckets [ 53 ] or using learned planning models to reflect detection quality [ 34 ] . However , they are either very coarse [ 53 ] or rely on optimized neural networks [ 34 ] , making it difficult to interpret and compare results in different settings . In this ∗Correspondence to bydeng @ waymo.com 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . paper , we take a novel approach to 3D object detection from an egocentric perspective . We start by reviewing the first principle : the detection quality relevant to the ego-agent ’ s planned trajectory , both at the moment and in the future , has the most profound impact on the ability to facilitate navigation . This leads us to transform detection predictions into two types of distance estimates relative to the ego-agent ’ s trajectory — lateral distance and longitudinal distance ( Fig . 1 ) . The errors on these two distances form our support distance error ( SDE ) concept , where the components can either be aggregated as the max distance estimation error or used independently , for different purposes . Compared to IoU , SDE ( as a shape metric ) is conditioned on the spatio-temporal relationship between the object and the ego-agent . Even a small mistake in detection near the ego-agent ’ s planned trajectory can incur a high SDE ( as in Fig . 2 left , object 3 ) . Additionally , SDE can be extended to evaluate the impact of detections to the ego-agent ’ s future plans ( for cases where an object comes close to the planned trajectory later in time ) . This is not feasible for IoU , which is invariant to the ego-agent position or trajectory ( shown in Fig . 2 ) . Using SDE to analyze a state-of-the-art detector [ 44 ] , we observe a significant error discrepancy between using a rectangular-shaped box ap- proximation and the actual object ’ s boundary , suggesting the need for a better representation to describe the fine-grained geometry of objects . To this end , we propose a simple lightweight refinement to box-based detectors named StarPoly . Based on a detection box , StarPoly predicts an amodal contour around the object , as a star-shaped polygon . Moreover , we incorporate SDE into the standard average precision ( AP ) metric and derive an SDE-based AP ( SDE-AP ) for conveniently evaluating existing detectors . In order to make an even more egocentric AP metric , we further add inverse distance weighting to the examples , obtaining SDE-APD ( D for distance weighted ) . With the proposed metrics , we observe different behaviors among several popular detectors [ 41 , 44 , 67 , 20 ] compared to what IoU-AP would reveal . For example , PointPillars [ 20 ] excels on SDE-AP in the near range in spite of its less competitive overall performance . Finally , we show that StarPoly consistently improves upon the box representation of shape based on our egocentric metric , SDE-APD . 2 Related Work . 3D Object Detection Modern LiDAR-based 3D object detectors can be organized into three sub-categories based on the way they represent the input point cloud : i.e. , voxelization-based detectors [ 55 , 8 , 21 , 50 , 19 , 60 , 47 , 68 , 58 , 20 , 64 , 56 ] , point-based methods [ 45 , 63 , 32 , 38 , 62 , 46 ] as well as hybrid methods [ 67 , 61 , 5 , 12 , 44 ] . Besides input representation , aggregating points across frames [ 13 , 65 , 14 , 41 ] , using additional input modalities [ 19 , 4 , 39 , 57 , 25 , 29 , 48 , 37 ] , and multi-task training [ 27 , 59 , 30 , 24 ] have also been studied to boost the performance . Despite such progress in model design , the output representation and evaluation metrics have remained mostly unchanged . Egocentric Computer Vision Egocentric vision has been studied in various applications . To name a few , understanding human actions from egocentric cameras , including action/activity recognition [ 9 , 35 , 28 , 49 , 36 , 51 , 15 , 52 ] , action anticipation [ 43 , 16 ] , and human object interaction [ 26 ] have been widely studied . Egocentric hand detection/segmentation [ 23 , 22 , 1 , 42 ] , and pose estimation [ 54 , 66 , 31 ] are among other applications . Arguably , 3D detection for autonomous driving can be naturally viewed as another egocentric application where data is captured by sensors attached to the car . However , classic IoU-based evaluation metrics ignore the egocentric nature of this application . 3D Object Detection Metrics Various extensions to the average precision ( AP ) metric have recently been proposed for the autonomous driving domain . nuScenes [ 2 ] consolidated mAP with more finegrained error types . Waymo Open Dataset [ 53 ] introduced mAP weighted by heading ( mAPH ) to reflect the importance of accurate heading prediction in motion forecasting . [ 34 ] proposed to examine detection quality from the planner ’ s perspective , by measuring the KL-divergence between future predictions conditioned on either noisy perception or ground truth . However , factors such as different planning algorithms or model training setups may cause this approach to yield inconsistent outcomes . Boundary-based Segmentation Metrics A different class of shape metrics on semantic segmentation masks evaluates the match quality of ground truth and predicted segmentation boundaries . Representative methods include Trimap IoU [ 3 , 18 ] , F-score [ 10 , 33 ] , boundary IoU [ 6 ] etc . These methods operate in an object-centric manner and do not take temporal information into consideration . 3 An Egocentric Shape Metric : Support Distance Error . Understanding the quality of modern 3D object detectors from an egocentric perspective is an underexplored topic and is open for new egocentric shape measures . In this section , we first look at the limitations of the box Intersection-over-Union ( IoU ) measure , the de facto choice to evaluate detection quality in popular benchmarks [ 11 , 53 , 2 ] and then introduce our newly propose egocentric shape metric : support distance error ( SDE ) . Limitations of box-based IoU IoU is an object-centric measure based on volumes ( or areas ) . As illustrated in Fig . 2 ( left ) , a prediction box with a relatively high IoU can still exhibit a high risk for an ego-agent ( the protruding box can cause the planner to brake suddenly , which in turn could lead to a tailgating collision ) . To understand such behavior at scale , we use collision detection as a “ gold standard ” to quantitatively reveal the limitation of IoUs . We select all the collisions reported by either the ground truth or a state-of-the-art PV-RCNN detector [ 44 ] in the validation set of the Waymo Open Dataset [ 53 ] . A ground truth collision is defined as an event where the object shape ( approximated by the aggregated object LiDAR points across all of its observations ) overlaps with the extended ego-agent shape ( approximated by a bounding box of the ego-agent , scaled up by 80 % ) . Collisions are estimated using detector boxes as the object ’ s shape . Table 1 presents the mean and median IoUs for true positive and false positive collision detections , whose difference is minimal , indicating that IoU is not effectively reflecting collision risk . Support Distance Error ( SDE ) In autonomous driving , one of the core uses of detection is to provide accurate object distance and shape estimates for motion planning ( which has collision avoidance as a one of the primary objectives ) . Instead of using box IoU , we can measure distances from the estimated shapes to the ego-agent ’ s planned trajectory . Specifically , we propose two types of distance measurements ( Fig . 1 ) 2 : • Lateral distance to an object : The minimal distance from any point on the object boundary to the line in the ego-agent ’ s heading direction . This distance is critical for the ego-agent to plan lateral trajectory maneuvers . • Longitudinal distance to an object : The minimal distance from any point on the object boundary to the central line perpendicular to the ego-agent ’ s heading direction . This distance is important to determine the speed profile and keep a safe distance from the objects in front . We use the term support distances for these two distance types , as they “ support ” the decision making in trajectory planning , and name the error between the ground truth support distance and the one estimated from a detector ’ s output as the support distance error ( SDE ) . We use SDElat to denote the lateral distance error and SDElon for the longitudinal error , and we define SDE as the maximum of the two . This formulation leads to two conceptual changes compared to IoU : we shift our focus from volume to boundary and from object-centric to ego-centric . This definition can also be extended to measure the impact of the detection quality on future collision risks . If we compute distances from the object boundary to the tangent lines at a future position ( at time t ) on the ego-agent ’ s trajectory , we can compute SDE for different future time steps ( denoted as SDE @ t ) . This is equivalent to measuring how close the object is to a future location of the ego-agent . To make the definition concrete , at time T = t , we assume the ego-agent ’ s pose is e ( t ) = ( x ( t ) , θ ( t ) ) , with x ( t ) ∈ R3 as its center ( e.g . the center of the ego-agent ’ s bounding box ) and θ ( t ) as its heading direction ( e.g . clock-wise rotating angle around the up-axis ) . We define the “ lateral line ” , the line crossing the ego-agent ’ s center and in the direction of its heading , as l ( t ) lat ; and the “ longitudinal line ” perpendicular to it as l ( t ) lon . On the other hand , we assume we have an object o and its predicted boundary is B ( o ) as a set of points on the boundary . The lateral/longitudinal distance of o at the current frame ( T = 0 ) is defined as : SDα = SDα ( B ( o ) , e ( 0 ) ) = minp∈B ( o ) d ( p , l ( 0 ) α ) , α ∈ { lat , lon } ( 1 ) where d computes the point-to-line distance . If the line passes through the object boundary the SDE would be 0 . Assume Bgt ( o ) is the object ground truth boundary , then the lateral/longitudinal support distance error is defined as : SDEα = SDα ( Bgt ( o ) , e ( 0 ) ) − SDα ( B ( o ) , e ( 0 ) ) , α ∈ { lat , lon } ( 2 ) The SDE sign has a physical meaning : positive errors mean the predicted boundary is protruding while negative means that a part of the object is not covered by the predicted boundary . For 2For simplicity , we define the distances from the object boundary to the ego-agent trajectory ( or the line perpendicular to it ) , instead of using the ego-agent shape , which varies across datasets and is typically not available in public datasets . simplicity , we take the absolute value of SDElat and SDElon by default and formally define SDE = max ( |SDElat| , |SDElon| ) , an aggregated value of both errors . To measure the impact of current frame detection quality on future plans , we define SDE @ t , which computes the SDE of an object t seconds in the future . Given the ground truth rigid motion R ( t ) of the object from T = 0 to T = t , we can transform its predicted boundary at frame T = 0 to its future position . In this way , the error patterns of the boundary can be consistently propagated into a future frame ( see Fig . 2 right for an example ) . The rigid motion can be derived between pairs of ground truth boxes of the object . We denote the transformed B ( o ) as B ( t ) ( o ) ′ . Note that it is different from the object shape prediction at time T = t : we are still measuring the quality of the T = 0 prediction , but within a future egocentric context . The future support distance can be formally defined as : SDα @ t = SDα ( B ( t ) ( o ) ′ , e ( t ) ) = minp∈B ( t ) ( o ) ′d ( p , l ( t ) α ) , α ∈ { lat , lon } ( 3 ) Similarly , we define SDEα @ t as the difference in SDα @ t between the ground truth and the predicted boundary , where α ∈ { lat , lon } . Then SDE @ t = max ( |SDElat @ t| , |SDElon @ t| ) . We use SDE and SDE @ 0s interchangeably unless otherwise noted . Metric implementation details To faithfully compute the support distance , we aggregate object surface points ( from Lidar ) across all frames , during which the object is observed ( which cover different viewpoints of the object ) as a surrogate shape to the ground truth . This allows us to effectively compute distances to the boundary without requiring costly object shape annotations/modeling . By default , we use the real driving trajectory . The same implementation applies when one would like to evaluate SDE by providing an arbitrary set of intended trajectories ( from a planner or simulation ) . Comparing SDE with IoU In Fig . 2 , we see SDElat is a highly useful indicator to reflect collision risk ( for object 3 ) . In Tab . 1 , we show that the mean and median SDE are sensitive shape measures and are inversely correlated with the collision risk . Naturally with larger t , SDE @ t increases , since the detections are based only on sensor data from the current frame T = 0 . Fig . 3 shows how SDE @ t and IoU change when we evaluate them at different time steps . Note that both SDE and SDE @ t are defined based on distances to the object boundary ( usually the part closer to the ego-agent ) . Clearly , better detection quality and boundary representation will result in an improved SDE metrics , which leads to the main idea of our next section .

Summary: This paper presents support distance error (SDE), a new evaluation metric 3D object detection from an egocentric point cloud. In contrast to Intersection over Union (IoU), SDE takes into account the potential collision induced by the object detector’s error over the planned trajectory of an ego-agent. It measures support distance (lateral and longitudinal directions) given the ego-agent’s orientation at given time. This metric is strongly indicative of collision risk.  Furthermore, the paper proposes StarPoly, a new object boundary representation that achieves better performance than traditional bounding box prediction on their new metrics SDE through experiments on autonomous driving Waymo Open dataset.