Revisiting 3D Object Detection From an Egocentric Perspective

1 Introduction . 3D object detection is a key problem in robotics , including popular applications such as autonomous driving . Common evaluation metrics for this problem , e.g . mean Average Precision ( mAP ) based on box Intersection-over-Union ( IoU ) , follow an object-centric approach , where errors on different objects are computed and aggregated without taking their spatiotemporal relationships with the egoagent into account . While these metrics provide a good proxy for downstream performance in general scene understanding applications , they have limitations for egocentric applications , e.g . autonomous driving , where detections are used to assist navigation of the ego-agent . In these applications , detecting potential collisions on the ego-agent ’ s trajectory is critical . Accordingly , evaluation metrics should focus more on the objects closer to the planned trajectory and to the parts/boundaries of those objects that are closer to the trajectory . Recent works have introduced a few modifications to evaluation protocols to address these issues , e.g. , breaking down the metrics into different distance buckets [ 53 ] or using learned planning models to reflect detection quality [ 34 ] . However , they are either very coarse [ 53 ] or rely on optimized neural networks [ 34 ] , making it difficult to interpret and compare results in different settings . In this ∗Correspondence to bydeng @ waymo.com 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . paper , we take a novel approach to 3D object detection from an egocentric perspective . We start by reviewing the first principle : the detection quality relevant to the ego-agent ’ s planned trajectory , both at the moment and in the future , has the most profound impact on the ability to facilitate navigation . This leads us to transform detection predictions into two types of distance estimates relative to the ego-agent ’ s trajectory — lateral distance and longitudinal distance ( Fig . 1 ) . The errors on these two distances form our support distance error ( SDE ) concept , where the components can either be aggregated as the max distance estimation error or used independently , for different purposes . Compared to IoU , SDE ( as a shape metric ) is conditioned on the spatio-temporal relationship between the object and the ego-agent . Even a small mistake in detection near the ego-agent ’ s planned trajectory can incur a high SDE ( as in Fig . 2 left , object 3 ) . Additionally , SDE can be extended to evaluate the impact of detections to the ego-agent ’ s future plans ( for cases where an object comes close to the planned trajectory later in time ) . This is not feasible for IoU , which is invariant to the ego-agent position or trajectory ( shown in Fig . 2 ) . Using SDE to analyze a state-of-the-art detector [ 44 ] , we observe a significant error discrepancy between using a rectangular-shaped box ap- proximation and the actual object ’ s boundary , suggesting the need for a better representation to describe the fine-grained geometry of objects . To this end , we propose a simple lightweight refinement to box-based detectors named StarPoly . Based on a detection box , StarPoly predicts an amodal contour around the object , as a star-shaped polygon . Moreover , we incorporate SDE into the standard average precision ( AP ) metric and derive an SDE-based AP ( SDE-AP ) for conveniently evaluating existing detectors . In order to make an even more egocentric AP metric , we further add inverse distance weighting to the examples , obtaining SDE-APD ( D for distance weighted ) . With the proposed metrics , we observe different behaviors among several popular detectors [ 41 , 44 , 67 , 20 ] compared to what IoU-AP would reveal . For example , PointPillars [ 20 ] excels on SDE-AP in the near range in spite of its less competitive overall performance . Finally , we show that StarPoly consistently improves upon the box representation of shape based on our egocentric metric , SDE-APD . 2 Related Work . 3D Object Detection Modern LiDAR-based 3D object detectors can be organized into three sub-categories based on the way they represent the input point cloud : i.e. , voxelization-based detectors [ 55 , 8 , 21 , 50 , 19 , 60 , 47 , 68 , 58 , 20 , 64 , 56 ] , point-based methods [ 45 , 63 , 32 , 38 , 62 , 46 ] as well as hybrid methods [ 67 , 61 , 5 , 12 , 44 ] . Besides input representation , aggregating points across frames [ 13 , 65 , 14 , 41 ] , using additional input modalities [ 19 , 4 , 39 , 57 , 25 , 29 , 48 , 37 ] , and multi-task training [ 27 , 59 , 30 , 24 ] have also been studied to boost the performance . Despite such progress in model design , the output representation and evaluation metrics have remained mostly unchanged . Egocentric Computer Vision Egocentric vision has been studied in various applications . To name a few , understanding human actions from egocentric cameras , including action/activity recognition [ 9 , 35 , 28 , 49 , 36 , 51 , 15 , 52 ] , action anticipation [ 43 , 16 ] , and human object interaction [ 26 ] have been widely studied . Egocentric hand detection/segmentation [ 23 , 22 , 1 , 42 ] , and pose estimation [ 54 , 66 , 31 ] are among other applications . Arguably , 3D detection for autonomous driving can be naturally viewed as another egocentric application where data is captured by sensors attached to the car . However , classic IoU-based evaluation metrics ignore the egocentric nature of this application . 3D Object Detection Metrics Various extensions to the average precision ( AP ) metric have recently been proposed for the autonomous driving domain . nuScenes [ 2 ] consolidated mAP with more finegrained error types . Waymo Open Dataset [ 53 ] introduced mAP weighted by heading ( mAPH ) to reflect the importance of accurate heading prediction in motion forecasting . [ 34 ] proposed to examine detection quality from the planner ’ s perspective , by measuring the KL-divergence between future predictions conditioned on either noisy perception or ground truth . However , factors such as different planning algorithms or model training setups may cause this approach to yield inconsistent outcomes . Boundary-based Segmentation Metrics A different class of shape metrics on semantic segmentation masks evaluates the match quality of ground truth and predicted segmentation boundaries . Representative methods include Trimap IoU [ 3 , 18 ] , F-score [ 10 , 33 ] , boundary IoU [ 6 ] etc . These methods operate in an object-centric manner and do not take temporal information into consideration . 3 An Egocentric Shape Metric : Support Distance Error . Understanding the quality of modern 3D object detectors from an egocentric perspective is an underexplored topic and is open for new egocentric shape measures . In this section , we first look at the limitations of the box Intersection-over-Union ( IoU ) measure , the de facto choice to evaluate detection quality in popular benchmarks [ 11 , 53 , 2 ] and then introduce our newly propose egocentric shape metric : support distance error ( SDE ) . Limitations of box-based IoU IoU is an object-centric measure based on volumes ( or areas ) . As illustrated in Fig . 2 ( left ) , a prediction box with a relatively high IoU can still exhibit a high risk for an ego-agent ( the protruding box can cause the planner to brake suddenly , which in turn could lead to a tailgating collision ) . To understand such behavior at scale , we use collision detection as a “ gold standard ” to quantitatively reveal the limitation of IoUs . We select all the collisions reported by either the ground truth or a state-of-the-art PV-RCNN detector [ 44 ] in the validation set of the Waymo Open Dataset [ 53 ] . A ground truth collision is defined as an event where the object shape ( approximated by the aggregated object LiDAR points across all of its observations ) overlaps with the extended ego-agent shape ( approximated by a bounding box of the ego-agent , scaled up by 80 % ) . Collisions are estimated using detector boxes as the object ’ s shape . Table 1 presents the mean and median IoUs for true positive and false positive collision detections , whose difference is minimal , indicating that IoU is not effectively reflecting collision risk . Support Distance Error ( SDE ) In autonomous driving , one of the core uses of detection is to provide accurate object distance and shape estimates for motion planning ( which has collision avoidance as a one of the primary objectives ) . Instead of using box IoU , we can measure distances from the estimated shapes to the ego-agent ’ s planned trajectory . Specifically , we propose two types of distance measurements ( Fig . 1 ) 2 : • Lateral distance to an object : The minimal distance from any point on the object boundary to the line in the ego-agent ’ s heading direction . This distance is critical for the ego-agent to plan lateral trajectory maneuvers . • Longitudinal distance to an object : The minimal distance from any point on the object boundary to the central line perpendicular to the ego-agent ’ s heading direction . This distance is important to determine the speed profile and keep a safe distance from the objects in front . We use the term support distances for these two distance types , as they “ support ” the decision making in trajectory planning , and name the error between the ground truth support distance and the one estimated from a detector ’ s output as the support distance error ( SDE ) . We use SDElat to denote the lateral distance error and SDElon for the longitudinal error , and we define SDE as the maximum of the two . This formulation leads to two conceptual changes compared to IoU : we shift our focus from volume to boundary and from object-centric to ego-centric . This definition can also be extended to measure the impact of the detection quality on future collision risks . If we compute distances from the object boundary to the tangent lines at a future position ( at time t ) on the ego-agent ’ s trajectory , we can compute SDE for different future time steps ( denoted as SDE @ t ) . This is equivalent to measuring how close the object is to a future location of the ego-agent . To make the definition concrete , at time T = t , we assume the ego-agent ’ s pose is e ( t ) = ( x ( t ) , θ ( t ) ) , with x ( t ) ∈ R3 as its center ( e.g . the center of the ego-agent ’ s bounding box ) and θ ( t ) as its heading direction ( e.g . clock-wise rotating angle around the up-axis ) . We define the “ lateral line ” , the line crossing the ego-agent ’ s center and in the direction of its heading , as l ( t ) lat ; and the “ longitudinal line ” perpendicular to it as l ( t ) lon . On the other hand , we assume we have an object o and its predicted boundary is B ( o ) as a set of points on the boundary . The lateral/longitudinal distance of o at the current frame ( T = 0 ) is defined as : SDα = SDα ( B ( o ) , e ( 0 ) ) = minp∈B ( o ) d ( p , l ( 0 ) α ) , α ∈ { lat , lon } ( 1 ) where d computes the point-to-line distance . If the line passes through the object boundary the SDE would be 0 . Assume Bgt ( o ) is the object ground truth boundary , then the lateral/longitudinal support distance error is defined as : SDEα = SDα ( Bgt ( o ) , e ( 0 ) ) − SDα ( B ( o ) , e ( 0 ) ) , α ∈ { lat , lon } ( 2 ) The SDE sign has a physical meaning : positive errors mean the predicted boundary is protruding while negative means that a part of the object is not covered by the predicted boundary . For 2For simplicity , we define the distances from the object boundary to the ego-agent trajectory ( or the line perpendicular to it ) , instead of using the ego-agent shape , which varies across datasets and is typically not available in public datasets . simplicity , we take the absolute value of SDElat and SDElon by default and formally define SDE = max ( |SDElat| , |SDElon| ) , an aggregated value of both errors . To measure the impact of current frame detection quality on future plans , we define SDE @ t , which computes the SDE of an object t seconds in the future . Given the ground truth rigid motion R ( t ) of the object from T = 0 to T = t , we can transform its predicted boundary at frame T = 0 to its future position . In this way , the error patterns of the boundary can be consistently propagated into a future frame ( see Fig . 2 right for an example ) . The rigid motion can be derived between pairs of ground truth boxes of the object . We denote the transformed B ( o ) as B ( t ) ( o ) ′ . Note that it is different from the object shape prediction at time T = t : we are still measuring the quality of the T = 0 prediction , but within a future egocentric context . The future support distance can be formally defined as : SDα @ t = SDα ( B ( t ) ( o ) ′ , e ( t ) ) = minp∈B ( t ) ( o ) ′d ( p , l ( t ) α ) , α ∈ { lat , lon } ( 3 ) Similarly , we define SDEα @ t as the difference in SDα @ t between the ground truth and the predicted boundary , where α ∈ { lat , lon } . Then SDE @ t = max ( |SDElat @ t| , |SDElon @ t| ) . We use SDE and SDE @ 0s interchangeably unless otherwise noted . Metric implementation details To faithfully compute the support distance , we aggregate object surface points ( from Lidar ) across all frames , during which the object is observed ( which cover different viewpoints of the object ) as a surrogate shape to the ground truth . This allows us to effectively compute distances to the boundary without requiring costly object shape annotations/modeling . By default , we use the real driving trajectory . The same implementation applies when one would like to evaluate SDE by providing an arbitrary set of intended trajectories ( from a planner or simulation ) . Comparing SDE with IoU In Fig . 2 , we see SDElat is a highly useful indicator to reflect collision risk ( for object 3 ) . In Tab . 1 , we show that the mean and median SDE are sensitive shape measures and are inversely correlated with the collision risk . Naturally with larger t , SDE @ t increases , since the detections are based only on sensor data from the current frame T = 0 . Fig . 3 shows how SDE @ t and IoU change when we evaluate them at different time steps . Note that both SDE and SDE @ t are defined based on distances to the object boundary ( usually the part closer to the ego-agent ) . Clearly , better detection quality and boundary representation will result in an improved SDE metrics , which leads to the main idea of our next section .

Summary: The paper proposed 1) a new metric SDE measuring the error in predicted lateral and longitude distance from the ego-car to another car, 2) an object contour prediction model to refine the shape from a simple box. 3) SDE-based AP and weighted AP for evaluating object detectors. The proposed SDE metric better reflects the risk of collision considering the ego-car trajectory.