Revisiting 3D Object Detection From an Egocentric Perspective
Authors: Boyang Deng
Charles R. Qi
Mahyar Najibi
Thomas Funkhouser
Yin Zhou
Dragomir Anguelov
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3d object detection
autonomous driving
egocentric
contour

Abstract: 3D object detection is a key module for safety-critical robotics applications such as autonomous driving. For these applications, we care most about how the detections affect the ego-agent’s behavior and safety (the egocentric perspective). Intuitively, we seek more accurate descriptions of object geometry when it’s more likely to interfere with the ego-agent’s motion trajectory. However, current detection metrics, based on box Intersection-over-Union (IoU), are object-centric and aren’t designed to capture the spatio-temporal relationship between objects and the ego-agent. To address this issue, we propose a new egocentric measure to evaluate 3D object detection,  namely Support Distance Error (SDE). Our analysis based on SDE reveals that the egocentric detection quality is bounded by the coarse geometry of the bounding boxes. Given the insight that SDE would benefit from more accurate geometry descriptions, we propose to represent objects as amodal contours, specifically amodal star-shaped polygons, and devise a simple model, StarPoly, to predict such contours. Our experiments on the large-scale Waymo Open Dataset show that SDE better reflects the impact of detection quality on the ego-agent’s safety compared to IoU; and the estimated contours from StarPoly consistently improve the egocentric detection quality over recent 3D object detectors.

Revisiting 3D Object Detection From an Egocentric Perspective
Boyang Dengi*
Charles R. Qit
Mahyar Najibi
Thomas Funkhouser
Yin Zhou
Dragomir Anguelov!
Waymo LLC
Google Research
Abstract
3D object detection is key module safety-critical robotics applications such autonomous driving. For such applications, care the most about how the detections impact the ego-agent's behavior and safety (the egocentric perspective). Intuitively, we seek more accurate descriptions of object geometry when it's more likely to interfere with the ego-agent's motion trajectory. However, current detec- tion metries, based on box Intersection-over-Uniiee not designed to capture the ppatio-tempore relationship between objects and the ego-agent. To address this issue, we propose a new egocentric measure evaluate 3D object detection: Support Distance Error (SDE). Our analysis based on SDE reveals that the egocentric detection quality bounded by the coarse geometry of the bounding boxes. Given the insight that SDE can be improved by more accu- rate geometry descriptions, we propose to represent objects as amodal contours, specifically amodal star-shaped polygons, and devise simple model, StarPoly, predict such contours. Our experiments on the large-scale Waymo Open Dataset show that SDE better reflects the impact of detection quality the ego-agent's safety compared to loU; and the estimated contours from StarPoly consistently improve the egocentric detection quality over recent 3D object detectors.
(IoU), are object-centric and are
Introduction
3D object detection is key problem in robotics, including popular applications such as autonomous driving. Common evaluation metrics for this problem, c.g. mean Average Precision (mAP) based on box Intersection-over-Uuu Union (IoU), follow object-centric approach, where errors different objects are computed and aggregated without taking their spatiotemporal relationships with the ego- agent into account. While these metrics provide a good proxy for downstream performance in general scene understanding applications, they have limitations for egocentric applications, e.g. autonomous driving, where detections used to assist navigation of the ego-agent. In these applications, detecting potential collisions on the ego-agent's 's trajectory critical. Accordingly, evaluation metrics should focus objects that are closer to the trajectory.
the objects closer the planned trajectory and to the parts/boundaries of those
Recent works have introduced few modifications to evaluation protocols to address these issues, e.g., breaking down the metrics into different distance buckets [53] or using learned planning models to reflect detection quality [34]. However, they are either very coarse [53] rely optimized neural networks [34], making it difficult to interpret and compare results in different settings. this
Cooresponndncce to bydengovaymo.com
35th Conference on Neural Information Processing Systems (NcurIPS 2021), virtual.

---

paper, we take novel approach to 3D object detection from an egocentric perspective. We start by reviewing the first principle: the detection quality relevant to the ego-agent's planned trajectory, both at the moment and in the future, has the most profound impact on the ability to facilitate navigation. This leads us transform detection predictions into two types of distance estimates relative to the ego-agent's s trajectory two distances form our support distance error (SDE) concept, where the components can either be aggregated as the max distance estimation error or used independently, for different purposes.
lateral distance and longitudinal distance (Fig. 1). The errors on these
Compared IoU, SDE (as a shape metric) is conditioned on the spatio-temporal relationship between the object and the ego-agent. Even a small mistake in detection near the ego-agent's planned trajectory can incur high SDE (as in Fig. 2 left, object 3). Additionally, SDE can be ex- tended to evaluate the impact of detections the ego-agent's 's future plans (for cases where ob- ject comes close to the planned trajectory later in time). This is not feasible for IoU, which is invari- ant to the ego-agent position or trajectory(shown in Fig. 2).
Heading
Lateral Distance
Ego- EgonAteet:
Longitudinal Distance
Object
Figure I: Lateral distance and longitudinal distance. These two types of support distance measure how far an object's shape boundary is to the observer (ego- agent) in both the direction along the observer velocity (longitudinal) and perpendicular to it (lateral).
Using SDE to analyze state-of-the-art detec- tor [44], we observe a significant error discrep- ancy between using a rectangular-shaped box ap- proximation and the actual object's ss boundary, suggesting the need for a better representation to describe the fine-grained geometry of objects. To this end, we propose a simple lightweight refine- ment to box-based detectors named StarPoly. Based on detection box, StarPoly predicts amodal contour around the object, as a star-shaped polygon.
Moreover, we incorporate SDE into the standard average precision (AP) metric and derive an SDE-based AP (SDE-AP) for conveniently evaluating existing detectors. order make an even more egocentric AP metric, we further add inverse distance weighting to the examples, obtaining SDE-APD (D for distance weighted). With the proposed metrics, we observe different behaviors among several popular detectors [41, 44, 67, 20] compared to what loU-AP would reveal. For example, PointPillars [20] excels on SDE-AP the near range in spite of its less competitive overall performance. Finally, we show that StarPoly consistently improves upon the box representation of shape based on our egocentric metric, SDE-APD.
Related Work
3D Object Detection Modern LiDAR-based 3D object detectors be organized into three sub-categories based on the way they represent the input point cloud: i.e., voxelization based detectors [55, 8, 21, 50, 19, 60, 47, 68, 58, 20, 64, 56), point-based methods [45, 63, 32, 38, 62, 46] as well as hybrid methods [67, 61, 5, 12, 44]. Besides input representation, aggregating points across frames [13, 65, 14, 41 using additional input modalities [19, 4, 39, 57, 25, 29, 48, 37], and multi-task training [27, 59, 30, 24] have also been studied to boost the performance. Despite such progress in model design, the output representation and evaluation metrics have remained mostly unchanged.
Egocentric Computer Vision Egocentric vision has been studied in various applications. To name a few, understanding human actions from egocentric cameras, including action/activity recognition [9, 35, 28, 49, 36, 51, 15, 52], action anticipation [43, 16]. and human object interaction [26] have been widely studied. Egocentric hand detectioon/eemmenttiion [23, 22, 1, 42], and pose estimation [54, 66, 31] are among other applications. Arguably, 3D detection for autonomous driving can be naturally viewed as another egocentric application where data captured by sensors attached the car. However, classic loU-based evaluation metrics ignore the egocentric nature of this application.
3D Object Detection Metrics Various extensions to the average precision (AP) metric have recently been proposed for the autonomous driving domain. nuScenes[2] consolidated mAP with more fine- grained error types. Waymo Open Dataset[53] introduced mAP weighted by heading (mAPH) to reflect the importance of accurate heading prediction in motion forecasting. [34] proposed to examine

---

Object 2:
Ego-Agent Trajectory
IoU=0.66
SDE
SDE 0.04m
Object 1601s:
1.9
0.91
Object 1:
SDE OD 0.m2
0.91 SDE
Ego Aeett
@ ls
Object
J.820.82
Ego-Agent
SD 3 0.42m
T
Os
T 1s
Figure 2: Illustration of IoUs lateral distances real scene. visualize the scene from bird's eye view where Lidar points are gray; green boxes are ground truth boxes; and red boxes are detector boxes. Left: We show that loU the high risk mistake of the object 3's box is not reflected by the high IoU. In contrast, while object 2 has lower IoU, box boundary accurately estimated, thus the impact to ego-agent planning limited. shown. compared to IoU, SDE is more indicative of the perception quality's 's impact on driving. Right: We show how SDE changes when evaluated at future time (colored in purple), reflecting how the current frame's 's perception quality influences decision making into the future. The detection box is transformed to future frame based on the rigid motion between the ground truth boxes at Os and by motion prediction). While object has low SDE at T on the left, emror significantly increases at 1s, as the box cannot capture the fine-grained geometry at the object corner (see the zoom in view).
object-centric measure is not directly reflecting the risk of collision (colored in blue)
1s (which excludes the error introduced
mloU v.8 CDA
mSDE CDA
CDA
0.940
0.145
TP Collision
FP/EN Collision
abd
Measure
0.077
0..922
0.137
Mean
Median
Mean
Median
0.970
0.906
CDA SEDe
0.130
IoU f
0.903
0.912
0.904
0.903
0.968
0.887
SDE
0.114
0.094
0.162
0.153
0.965
0.114
Time (4)
Tmme (6)
Distributions of error measures in two types of collision detection cases. In "TP Collision", both the ground truth points the prediction report a collision. In "FP/EN Collision", either the ground truth (FN) or the prediction (FP) reports a collision. While the distributions of IoU in TP and FP/FN are higher mean IoU in FP/EN, SDEs among TP are clearly better than FP/FN with an im- provement of 30% in mean and 40% in median.
Table 1:
Figure 3: Correlations with collision detection accu- racy (CDA). For evaluation moments from prediction time to 10s in the future, we compute the CDA, mean IoU (mIoU), and mean SDE (mSDE). We see from the curve that mloU is not correlated with the accuracy drop ToUs don't vary with ego motions; while mSDE inversely correlated to collision detec- tion accuracy due to its egocentric nature.
close with
detection quality from the planner's perspective, by measuring the KL--divergence between future predictions conditioned on either noisy perception or ground truth. However, factors such as different planning algorithms or model training setups may cause this approach to yield inconsistent outcomes.
Boundary-based Segmentation Metrics A different class of shape metrics on semantic segmen- tation masks evaluates the match quality ground truth and predicted segmentation boundaries. Representative methods include Trimap IoU[3, 18), F-score[ [10, 33), boundary [oU[6] etc. These methods operate in an object-centric manner and do not take temporal information into consideration.
An Egocentric Shape Metric: Support Distance Error
Understanding the quality of modern 3D object detectors from an egocentric perspective is an under- explored topic and open for new egocentric shape measures. this section, we first look at (IoU) measure, the de facto choice to evaluate
the limitations of the box inteerecttinnover-Unnie detection quality in popular benchmarks [11, 1, 53, 2] and then introduce our newly propose egocentric shape metric: support distance error (SDE).

---

Limitations of box-based IoU IoU is an object-centric measure based on volumes (or areas). As illustrated in Fig. 2 (left). a prediction box with a relatively high IoU can still exhibit a high risk for an ego-agent (the protruding box can cause the planner to brake suddenly, which in turn could lead to a tailgating collision).
To understand such behavior at scale, we use collision detection as a "gold standard" to quantitatively reveal the limitation of IoUs. We select all the collisions reported by either the ground truth or state-of-the-art PV-RCNN detector [44] in the validation set of the Waymo Open Dataset [53]. A ground truth collision is defined as an event where the object shape (approximated by the aggregated object LiDAR points across all of its observations) overlaps with the extended ego-agent shape (approximated by a bounding box of the ego-agent, scaled up by 80%). Collisions are estimated using detector boxes as the object's is shape. Table 1 presents the mean and median IoUs for true positive and false positive collision detections, whose difference is minimal, indicating that IoU is not effectively reflecting collision risk.
Support Distance Error (SDE) autonomous driving, one of the core uses of detection is to provide accurate object distance and shape estimates for motion planning (which has collision avoidance as one of the primary objectives). Instead of using box IoU, we can measure distances from the estimated shapes to the ego-agent's planned trajectory. Specifically, we propose two types of distance measurements (Fig. 1)²:
Lateral distance to an object: The minimal distance from any point on the object boundary to the line in the ego-agent's 's heading direction. This distance is critical for the ego-agent to plan lateral trajectory maneuvers.
Longitudinal distance to an object: The minimal distance from any point on the object boundary to the central line perpendicular to the ego-agent's heading direction. This distance is important to determine the speed profile and keep a safe distance from the objects in front.
We use the term support distances for these two distance types, as they "support" the decision making in trajectory planning, and name the error between the ground truth support distance and the one estimated from a detector's 's output as the support distance error (SDE). We use SDE to denote the lateral distance error and SDE for the longitudinal error, and we define SDE as the maximum of the two. This formulation leads to two conceptual changes compared to IoU: we shift our focus from volume to boundary and from object-centric ego-centric.
This definition can also be extended to measure the impact of the detection quality on future collision risks. If we compute distances from the object boundary to the tangent lines at future position (at time t) on the ego-agent's trajectory, we can compute SDE for different future time steps (denoted as SDE@t). This is equivalent to measuring how close the object is to a future location of the ego-agent.
To make the definition concrete, at time T = t, we assume the ego-agent's 's pose e(t) (zx )(1)) with (! E R2 as its center (e.g. the center of the ego-agent's bounding box) and g(0) as its heading direction (e.g. clock-wise rotating angle around the up-axis). We define the "lateral line". the line crossing the ego-agent's center and in the direction of its heading, as tat and the "Iongitudinal line" perpendicular to it as 1(0) On the other hand, we assume we have an object and its predicted boundary is (()) as a set of points on the boundary. The lateral/longitudinal distance of o at the current frame (T = 0) is defined as:
(I) where d computes the point-to-line distance. If the line passes through the object boundary the SDE would be O. Assume (o) is the object ground truth boundary, then the lateral/longitudinal support distance error is defined as:
SD SD, (0) mi (() E {lat,lon)
SDE, SD. (By) e0) (B(o), a {lat, lon]
The SDE sign has physical meaning: positive errors mean the predicted boundary is protruding while negative means that a part of the object is not covered by the predicted boundary. For
For simplicity, we detine the distances from the object boundary to the ego-agent trajectory (or the line perpendicular to it), instead of using the ego- ago-agent shape, which varies across datasets and is typically not available in public datasets.

---

Box
Prod/ CCVC
CT/Box
CT Points
Input Points
SDE. 0.65m
SDE 3.35m
SDE
0.39m
0.51m
(a) Box Missligned (loU=0.66)
(b) Wrong Box Size
(c) Box with High IoU
(d) Box with High ToU
(loU=0.73)
(IoU=0.91)
(IoU=0.95)
Figure 4: Failure cases with large SDEs (>= 0.3m). (a) and (b): The detector boxes are poorly aligned with the ground truth either in orientation or size. (c) and (d): The detector boxes yield near-perfect IoUS with the ground truths but still incur high SDE. The convex visible contours (CVC) are derived based the input points within detection at the current frame. Note that SDEs here computed against temporally aggregated Lidar points (the GT Points) and IoUs are computed between detections and ground truth boxes.
simplicity, we take the absolute value of SDE and SDEton by default and formally define SDE max((SDFzar): (SDE l), an aggregated value of both errors.
To measure the impact of current frame detection quality on future plans, define SDE@t, which computes the SDE of object t seconds in the future. Given the ground truth rigid motion R of the object from = 0 to T = f, we can transform its predicted boundary at frame 7 = to its future be consistently propagated into future
position. In this way, the error patterns of the boundary frame (see Fig. 2 right for an example). The rigid motion can be derived between pairs of ground truth boxes of the object. We denote the transformed (o)) as (o)' Note that it is different from the object shape prediction at time T = t: are still measuring the quality of the T prediction, but within a future egocentric context. The future support distance can be formally defined as:
SD @1 = (D. el
min mip 1( ) a E {lat, Ion]
Similarly, we define SDE,, @t as the difference in SD, @t between the ground truth and the predicted boundary, where flat, lon). Then SDE@t max([SDE @tl, [SDE @t)). We use SDE and SDE@0s interchangeably unless otherwise noted.
Metric implementation details To faithfully compute the support distance, we aggregate object surface points (from Lidar) across all frames, during which the object is observed (which cover different viewpoints of the object) as surrogate shape to the ground truth. This allows us to effectively compute distances to the boundary without requiring costly object shape annotations/moodeingg By default, we use the real driving trajectory. The same implementation applies when one would like to evaluate SDE by providing an arbitrary of intended trajectories (from planner or simulation).
Comparing SDE with IoU Fig. 2, we see SDE is a highly useful indicator to reflect collision risk (for object 3). In Tab. 1, we show that the mean and median SDE are sensitive shape measures and are inversely correlated with collision risk. Naturally with larger I, SDE@t increases, since the detections are based only on sensor data from the current frame T O. Fig. 3 shows how SDE@t and loU change when we evaluate them at different time steps.
Note that both SDE and SDE@t t are defined based on distances to the object boundary (usually the part closer to the ego-agent). Clearly, better detection quality and boundary representation will result in an improved SDE metrics, which leads to the main idea of our next section.
Shape Representations and SDE
In this section, we use SDE to analyze detection quality safety-critical scenarios and highlight the importance of the shape representation therein. We further propose a new amodal contour representation and neural network model (StarPoly) for contour estimation and demonstrate produces significant SDE improvements.
5

---

Box
PV-RCNN
Ground Truth
0.20
CVC
Box
CVC
StarPoly
CVC
StarPoly
StarPoly
mSDE of [0mn, 5mi)
0.107
0.090
0.063
0.059
0.083
0.046
0.15
mSDE of [5m, 10m)
0.087
0.064
0.070
0.076
0.053
0.10
mSDE of [10m, 20m)
0.140
0.155
0.086
0.094
0.142
0.068
mSDE [20m, 40m)
0.207
0.206
0.152
0.132
0.235
0.105
10
Time From Now (s)
Table 2: Comparing mean SDE (mSDE) boxes, convex vis- ible contours (CVC), and our StarPoly at different distance ranges. While lower than mSDE of detector box, CVC's SDE rises rapidly towards far ranges. Mcanwhile, StarPoly is superior than both box and CVC in all ranges.
Figure 5: mSDE [0m, 10m) at different
time steps. CVC's mSDE significantly in- creases as the evaluation goes into the fu- ture. In contrary. StarPoly consistently out- performs others.
4.1 Qualitative Analysis of Bounding Box Failure Cases
To understand how detector boxes perform under the SDE measure, we select PV-RCNN [44], a top- performing single-frame point-cloud-base detector in popular autonomous driving benchmarks [11, 53]), for our analysis. All analysis is based on the Waymo Open Dataset [53] validation set. Fig. 4 illustrates some representative failure cases among the detector boxes, with high SDE.
We find that even when a box aligns reasonably well with the ground truth it can still incur high SDE. By comparing the predicted detection against the point cloud inside in the box, notice that rectangular boxes typically do not tightly surround the object boundary. In particular, the discrepancy between box corners and the actual object boundary contributes a considerable amount of SDE. This observation inspires us to seek more effective representations of the fine-gratn mne-grained object geometry.
4.2 Convex Visible Contours
An intuitive solution to obtain a tighter object shape fit by leveraging the Lidar points. Specifically, one can extract all points within the detector box (after removing points on the ground) and compute their convex hull, as a convex visible contour (CVC). In contrast to amodal object shape, CVC is computed only from the visible Lidar points at the current frame. Fig. 4 provides some visualizations.
Tab. shows how CVC compares with bounding boxes SDE. Considering that CVC is heavily dependent on the quality of the box it resides in, we also evaluate CVC directly based on the ground truth boxes, which can be seen as the upper bound for CVC (col. 6). We see that at near range, CVC can significantly improve SDE compared to the detector boxes (col. 2 vs 3). However, its effectiveness degrades at longer ranges (col. 2 vs 3) and its performance inferior to ground truth boxes (col. 5 vs 6). We hypothesize that this is because CVC is vulnerable to occlusions, clutter and object point cloud sparsity at longer ranges, which are ubiquitous phenomena real world data. In Fig. 5, the analysis based on SDE@ @t confirms that CVC performs better than detector boxes the current frame but generalizes poorly to longer time horizons. To improve it, we need a representation that provides good coverage of both the visible and the occluded object parts.
4.3 Amodal Contour Estimation with StarPoly
We propose to refine box-based detection with amodal contours, polyline contour that covers the entire object shape (See Fig. 6 for an illustration). Our model, StarPoly, implements contours as star-shaped polygons and predicts amodal shape via a neural network 3 It can be employed to refine predicted boxes for any off-the-shelf detectors.
Input The input the StarPoly model is normalized object point cloud. We crop the object point cloud from its (extended) detection box. The point cloud is canonicalized based on the center and the heading of the detection box. as well as scaled by a scaling factor, s, such that the length of the longest side of the predicted box becomes 1.
AAthhough there are previous works on shape reconstucctinn/compttioo [7, 30), they are often trained on synthetic data and are not directly applicable real Lidar data. We leave more studies in designing the best contour estimation model future work and evaluate StarPoly as baseline towards better egocentric detection.

---

Parameterization As shown in Fig. 6, the star-s polygon is defined by a center point, h. and a list of vertices on its boundary. (V1, ..., vo), where n is the total number of vertices determining the shape resolution. We assume h is the center of the predicted box and sort (V1, Vv,) in clockwise order so that connecting the vertices successively produces a polygon. We constrain ey to have only 1 degree of freedom by defining 3/2 = c,d,, where (d) dm) is a list of unit vectors in predefined directions. Consequently, predicting star-shaped polygon is equivalent predicting (C1, Cn), for which we employ a PointNet [40] model (see the supplementary material for details).
Optimization Since ground truth contours are not available in public datasets, directly training the regression of (c1..., Cx) is infeasible. We resort to a surrogate objective for supervision. The objective combines three intuitive goals, namely coverage, accuracy, and rightness. The coverage loss encourages the prediction to encompass all ground truth object points (aggregated points in the object bounding box from all frames which the object appears, with ground points removed). Moreover, as the input point cloud already reveals part of the object boundary visible to the ego-agent, the accuracy loss requires the prediction to fit the visible boundary as tight as possible. On the other hand, the tightness loss minimizes the area of the predicted contour. The combination of these three goals leads to the reconstruction of contours without requiring ground truth contour supervision. More formally, the coverage loss Le, the accuracy loss La. the tightness loss L,, and consequently the overall objective for one ground truth point cloud X are defined as follows:
C
1
max
I X Up
X
1,0)
Encompass all object points, Ce.
X
U1 X Ur
1t X U,
zEX
B
T Vy
1j X
Fit tight to visible boundaries, Ca-
(4)
B
U1 X Cr
1) X Ur
mEX
7
Σ
||:
Minimize the area of contours, L.-
where x is a point from X, y is a weight parameter for Lt, and X represents cross product. Note that in Le, I and r are selected so that di and d, span a wedge shape containing æ (as shown in Fig. 6). Intuitively, Le is computing the barycentric coordinates of x with regard to and u, within the triangle 2 Ahtre, and encouraging x to be on the same side as regarding ey0, - the necessary and sufficient condition for x E Ahvyu, Similarly, La is forcing the points on the visible boundary B to be on the predicted boundary as well. Meanwhile, La is pulling all v; towards h.
d d
Figure 6: StarPoly formulation.
Results In Tab. 2 we see that updating the bounding box output of PV-RCNN to StarPoly contours significantly improves the mean SDE under all distance buckets (e.g. at 0-5m, it improves from 10.7cm to 8.6cm, which is around 20% error reduction). Similar improvements also appear on the ground truth boxes (col. 4-6). In Fig. 5, we also show how StarPoly improves on SDE@t. In all time steps, the StarPoly has lower SDE than both bounding boxes and visible contours, showing its advantage of getting the best of both worlds.
5 Egocentric Evaluation of 3D Object Detectors
In this section, we incorporate SDE into the standard average precision (AP) metric and evaluate various detectors and shape representations the Waymo Open Dataset [53].
SDE-AP: Detection AP based on the SDE shape metric To compare different detectors on their
egocentric performance, we cannot just use the SDE measure, which does not consider false positive (FP) and false negative (FN) cases. Therefore, we propose to adapt the traditional loU-based AP (IoU-AP) to an SDE-bbased one (SDE-AP). Specifically, we replace the classification criterion for true positives (TP) from IoU to an SDE-based threshold and use SDE 20cm as the threshold (see the
7

---

+V-MVF++
MVF++
PV-RONN
PointPillars
[0m, 5m]
[5m, 10m]
110m, 20m]
[20m, 40m]
1,00
1,00
1.00
1.00
0.90
0.50
0.80
0.80
0.80
0.50
0.00
0.60
0.60
0.60
0.50
0.50
0.50
0.30
IPU-AP
SDE-AP
1PP-AP
SDE AP
IPU-AP
SDE-AP
IcU-AP
SDE-AP
Figure 7: Distance breakdowns of IoU-AP and SDE-AP. SDE-AP can better differentiates different egocentric detection quality than loU-AP, especially in the near ranges ([0m, 5m] and [5m, 10m]).
Supplementary material for more why selected this number). In addition, use SDE-based criterion (instead of an loU-based one) to match predictions and ground truth.
SDE-APD: inverse distance weighted SDE-AP Although SDE-AP is based on the egocentric SDE measure, it weights objects at various distances from the ego agent trajectory equally. To design more strongly egocentric measure, further propose variant of the SDE-AP with inverse-distance weighting, termed SDE-APD (the suffix means distance weighted). Specifically, for a given frame we have detections B {br}, i 1, ..., N and ground truth objects G {g:}
1, ..... M. We denote the matched ground truth object for b, as g(b:) E G. A prediction is counted as a true positive if (b, 9 (b,); e) 8 where ö is the SDE threshold and e the ego-agent pose. Then define the set of true positive predictions as TP { S DED (bb,, (b,);:) e) < 5} and false positive predictions as FP- B TP. The inverse distance weighted TP count (IDTP), FP count (IDFP) and ground truth count (IDG) for the frame are:
IDTP
Σ 1/d b.ETP
IDFP-
E1/g Σ 1/d IDG b.EFP
(5)
g(b.)
9.EG
where d the Manhattan distance from the prediction shape center to the ego-agent center and is hyper parameter controlling how much we focus the close-by objects (we set B 3, see the supplementary for more details).
The inverse distance weighted precision and recall are defined as IDTP/ + IDFP) and IDTP/I DDG respectively, both remain within [0, 1]. The SDE-APD the area under the PR-curve. Similar to SDE, which is defined both for the current frame and for future frames (SDE@t), the SDE-AP and SDE-APD metric also have future equivalents SDE-AP@1 and SDE-APD @t that can evaluate the impact of current frame perception on future plans.
5.1 Comparing Different Detectors on SDE-AP and SDE-APD
In this subsection, compare few representative point-cloud- based 3D object detectors on the SDE-AP and SDE-APD met- rics. We study several popular detectors: PointPillars [20], light-weight and simple detector widely used as a baseline; PV-RCNN [44], state-of-the -he--------------- detector with a sophisticated feature encoding: MVF++ [67, 41] (an improved version of the multi-view fusion detector), recent pop-rerfornnng detector; and finally 5T-MVF++ [41], extended version of MVF++ taking point clouds from consecutive frames as input, the most powerful among all.
Method
SDE-APD
loU-AP
5F-MVT+
0.874
0.863
MVF++
0.831
0.814
PV-RCNN
0.808
0.797
PointPillars 0.817
0.720
Table 3: SDE-APD and IoU-AP' different detectors.
Fig. shows the SDE-AP with distance breakdowns for all detectors and Table 3 shows the egocentric SDE-APD metric. An interesting observation from Fig. 7 about IoU-AP is that, while the four detectors have fairly close IoU-APs at close ranges (e.g. [0m, 5m)). we see significant gaps among them at longer ranges (e.g. [20m, 40m]). Since there are more objects longer ranges, those long-range buckets typically dominate the overall IoU-AP. In contrast, the SDE-AP is consistently more discriminative of the detectors especially for the very short range
'The loU-AP is compute using euclidean distance matching and 2D loU 0.7 the threshold.

---

0.95
0.03
Starfoy
CVC
SDE-APD APD
0.9
5K-MVF +
SDE-AI APD
0.9
MVF+
0.8
SDE APD
0.91
2
6
10
2
6
10
Box
Time
0.88
D.8
0,5
PV-RCNN
SDE-APD
StarPoly
PointPillars
0.84
0.9
0.0
0,3
0.7
0.8
0.80
2
10
2
10
+V+ + PV-RCNN PointPillars
Time
Time (9)
Figure 8: SDE-APD of detectors with different out- put representations (T=0s).
Figure 9: SDE-APD@ of boxes and StarPoly based on different detectors.
Box
CVC
StarPoly
3
Box: 0.317m CVC: 0.017m
Box: 0.345m CVC: 0.009m
StarPoly: 0.023m
StarPoly: 0.040m
2
Box: 0.5266 CVC: 0.009m
3
Box: 0,289m
CVC: 0.027m
SturPoly: 0.067m
StarPoly: 0.015m
Figure 10: Qualitative Results. A scene from the validation set of the Waymo Open Dataset with StarPoly predictions shown green contours. We also zoomed in into 4 vehieles closest to the ego-agent and compared StarPoly (green) with predicted box (red), and CVC (blue). SDE,.: are reported under each zoom-in.
in [0m, 5m]. We even see some change of rankings PointPillars, with the lowest overall loU-AP, outperforms PV-RCNN and MVF++ close-range [Om, 5m] SDE-AP, suggesting it has a particularly strong short-range performance. This also implies that simply examining IoU-AP for selecting detectors can sub-optimal and our SDE-AP can provide an informative alternative perspective.
5.2 Comparing Various Shape Representations
In this subsection, we evaluate how detector output representations affect the overall detection performance in terms of SDE-APD evaluated at the current frame as well as into the future.
StarPoly implementation details. For the encoding neural network, we use the standard Point- Net [40] architecture followed by fully-connected layer to transform latent features to (ct,..., C.)..... 256 for all following experiments. (d1,... d,) uniformly sampled from
We use resolution the boundary of a square. During training, 7 and / are both set to 0.1, which is determined by grid search over the hyperparameters. Please refer to the supplementary material for more model details.
Results In Fig. 8, we compare the egocentric performance of different representations using SDE- APD. StarPoly consistently improves the egocentric result quality across the different detectors. Interestingly, StarPoly largely closes the gap between the different detectors, reducing the difference between 5F-MVF++ [41] and PV-RCNN by factor of 3. This implies that StarPoly's 's amodal contours can greatly compensate for the limitations of the initial detection boxes, especially of those with poorer quality. StarPoly also outperforms convex visible contours (CVC) across all detectors. In Fig. 9, we evaluate the performance of the different representations at future time steps. We observe that StarPoly remains superior to the detector boxes across all time steps, differentiating from the convex visible contours that decay catastrophically over time (shown in Fig. 5). Fig. 10 shows a scene with StarPoly amodal contours estimated for all vehicles. The zoom-in figures reveal how amodal contours have more accurate geometry, and lower SDE, than both boxes and visible contours. However, they are not yet perfect, especially on the occluded object sides. Improving contour estimation even further is promising direction for future work.

---

6 Conclusion
In this paper, we propose egocentric metrics for 3D object detection, measuring its quality in the current time step, but also its effects on the ego agent's plans in future timesteps. Through analysis, we have shown that our egocentric metries provide valuable signal for robotic motion planning applications, compared to the standard box intersection-over--miie criterion. Our metrics reveal that the coarse geometry of bounding boxes limits the egocentric prediction quality. To address this, we have proposed using amodal contours as a replacement to bounding boxes and introduced StarPoly, simple method predict them without direct supervision. Extensive evaluation on the Waymo Open Dataset demonstrates that StarPoly improves existing detectors consistently with respect to our egocentric metrics.
Acknowledgements and Funding Transparency Statement. We thank Johnathan Bingham and Zoey Yang for the help on proofreading our drafts, and the anonymous reviewers for their constructive comments. All the authors are full-time employees of Alphabet Inc. and are fully-funded by the subsidairies of Alphabet Inc., Google LLC and Waymo LLC. All experiments are done using the resources provided by Waymo LLC.
References
Sven Bambach, Stefan Lee, David Crandall, and Chen Yu. Lending hand: Detecting hands and recognizing activities complex egocentric interactions. Proceedings of the IEEE International Conference on Computer Vision, pages 1959-1957, 2015.
[2] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[3] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, convolution, and fully connected erfs. IEFE Transactions on Pattern Analysis and Machine Intelligence, PP, 2016.
[4] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In CVPR, 2017.
[5] Y. Chen. S. Liu, X. Shen, and
Fast point r-cnn. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7774-9783, 2019.
[6] Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C. Berg, and Alexander Kirillov. Boundary IoU: Improving object-centric image segmentation evaluation. In CVPR, 2021.
[7] Angela Dai, Charles Ruizhongtai Qi, and Matthias Nießner. Shape completion using 3d-encoder-ppediccor cnns and shape synthesis. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017.
[8] Martin Engeleke, Dushyant Rao, Dominic Zeng Wang. Chi Hay Tong, and Ingmar Posner. Vote3deep: Fast object detection in point clouds using efficient convolutional neural networks. In ICRA. IEEE, 2017.
[9] Alireza Fathi, Xiaofeng Ren, and James M Rehg. Learning to recognize objects in egocentric activities. In CVPR 2011, pages 3281 --288. IEEE, 2011.
[10] Florent Perronnin (Xerox (XRCE) Grenoble) Gabriela Csurka (Xerox Research Centre Europe Diane Lar- lus. What is a good evaluation measure for semantic segmentation? In Proceedings of British Muchine Vision Conference. BMVA Press, 2013.
[11] Andreas Geiger, Philip Lenz, and Raquel Urtasun, Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
[12] Chenhang Hc, Hui Zeng, Jianqiang Huang, Xian-Sheng Hua, and Lei Zhang. Structure aware single-stage 3d object detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[13] Peiyun Hu, Jason Ziglar, David Held, and Deva Ramanan. What you see what you get: Exploiting visibility for 3d object detection. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[14] Rui Huang, Wanyue Zhang, Abhijit Kundu, Caroline Pantofaru, David A. Ross, Thomas A. Funkhouser, and Alireza Fathi. An LSTM approach temporal 3d object detection in lidar point clouds. CoRR, 2020.
[15] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages
2019.
10

---

[16] Qiuhong Ke, Mario Fritz, and Bernt Schiele. Time-conditioned action anticipation in one shot. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9925-9934, 2019.
[17] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6888 41266980, 2014.
[18] Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected erfs with gaussian edge potentials. In Shawe-Taylor, R. Zemel, Bartlett, F. Pereira, and Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
[19] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven Waslander, Joint 3d proposal generation and object detection from view aggregation. In IROS. IEFF, 2018.
[20] Alex H Lang. Sourabh Vora, Holger Caesar. Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR, 2019.
[21] Bo Li. 3d fully convolutional network for vehicle detection in point cloud. arXiv preprint 161.0996,,. 2016.
[22] Cheng Li and Kris M Kitani. Model recommendation with virtual probes for egocentric hand detection. In Proceedings of the IEEE International Conference Computer Vision, pages 2624-2631, 2013.
[23] Cheng Li and Kris M Kitani. Pixel-level hand detection in ego-centric videos. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3570-3577, 2013.
[24] M. Liang, B. Yang, Y. Chen, R. Hu, and R. Urtasun. Multi-task multi-sensor fusion for 3d ohject detection. In 2019 IEEF/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pages 7337-7345, 2019.
[25] Ming Liang. Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi-sensor 3d object detection. In ECCV, pages 641-656, 2018.
[26] Miao Liu, Siyu Tang, Yin Li, and James M Rehg. Forecasting human-object interaction: joint prediction of motor attention and actions in first person vidco. In European Conference on Computer Vision, pages 704-721. Springer, 2020,
[27] Wenjie Luo, Bin Yang, and Raquel Urtasun. Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 2018.
[28] Minghuang Ma, Haoqi Fan, and Kris M Kitani. Going deeper into first-person activity recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1894-1903, 2016.
[29] G. P. Meyer. J. Charland. D. Hegde, A. Laddha. and C. Vallespi-Gonzaler Sensor fusion for joint 3d object detection and semantic segmentation. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1230-1237, 2019.
[30] Mahyar Najibi, Guangda Lai, Abhijit Kundu, Zhichao Lu, Vivek Rathod, Thomas Funkhouser, Caroline Pantofaru, David Ross, Larry Davis, and Alireza Fathi, Dops: learning to detect 3d objects and predict their 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11913 11922, 2020.
[31] Evonne Ng, Donglai Xiang, Hanbyul Joo, and Kristen Grauman. You2me: Inferring body pose in egocentric video via first and second person interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9890-9999999 990009990,20000 2020.
[32] Jiquan Ngiam, Benjamin Caine, Wei Han, Brandon Yang. Yuning Chai, Pei Sun, Yin Zhou, Xi Yi, Ouais Alsharif, Patrick Nguyen, Zhifeng Chen, Jonathon Shlens, and Vijay Vasudevan. Starnet: Targeted computation for object detection point clouds. CORR, 2019.
[33] F. Perazzi, Pont-Tuset, B. MeWilliams, L. Van Gool, M. Gross, and A. Sorkine- Hormung A benchmark dataset and evaluation methodology for video object segmentation. In 2016 IEEE Conference on Computer Vision and Patiern Recognition (CVPR), pages 724-732, 2016.
[34] Jonah Philion, Amlan Kar, and Sanja Fidler. Learning to evaluate perception models using planner-centric metrics. In /EEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[35] Hamed Pirsiavash and Deva Ramanan. Detecting activities of daily living in first-person camera views. In 2012 IEEE conference on computer vision and pattern recognition, pages
IEEE, 2012.
[36] Rafael Possas, Sheila Pinto Caceres, and Fabio Ramos. Egocentric activity recognition on a budget. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3967-5976 2018.
11

---

[37] Charles R Qi. Xinlei Chen, Or Litany, and Leonidas Guibas. Invotenet: Boosting 3d object detection in point clouds with image votes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4413, 2020.
[38] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In Proceedings of the IEEE International Conference Computer Vision, pages 9277-9286 2019.
[39] Charles R Qi. Wei Liu. Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgb-d data. In CVPR, 2018.
[40] Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017.
[41] Charles R Qi. Yin Zhou, Mahyar Najibi, Pei Sun, Khoa Vo, Boyang Deng, and Dragomir Anguelov. Offboard 3d object detection from point cloud sequences. CVPR, 2021.
[42] Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey. Understanding human hands in contact internet scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 9869-9878, 2020.
[43] Yang Shen, Bingbing Ni, Zefan Li, and Ning Zhuang. Egocentric activity prediction via event modulated attention. In Proceedings of the European Conference on Computer Vision (ECCV), pages 197 2922 2018.
[44] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn: Point-voxe voxel feature abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10529-10538, 2020.
[45] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. arXiv preprint arXiv: 1812.04244, 2018.
[46] Weijing Shi and Ragunathan (Raj) Rajkumar. Point-gnn: Graph neural network for 3d object detection in a point cloud. In The IEEE Conference on Computer Vision and Pattern Recognition ( (CVPR), June 2020.
[47] Martin Simon, Stefan Milz, Karl Amende, and Horst-Michael Gross. Complex-yolo: An euler-region- proposal for real-time 3d object detection on point clouds. In ECCV, 2018.
[48] Vishwanath A Sindagi, Yin Zhou, and Oncel Tuzel. Mvx-net: Multimodal voxelnet for 3d object detection. arXiv preprint arXiv: 1904.0944..00 904400169,, 2019.
[49] Suriya Singh, Chetan Arora, and CV Jawahar. First person action recognition using deep learned descriptors. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pages 2620-226622 2628, 2016.
[50] Shuran Song and Jianxiong Xiao. Deep sliding shapes for amodal 3d object detection in rgb-d images. In CVPR, 2016.
[51] Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. Lsta: Long short-term attention for egocen- tric action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9954-9963, 2019.
[52] Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. Gate-shift networks for video action recogni- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1102-1111, 2020.
[53] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2446-2454, 2020.
[54] Denis Tome, Patrick Peluse, Lourdes Agapito, and Hernan Badino, xr-egopose: Egocentric 3d human pose from an hmd camera. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7728-7738, 2019.
[55] Dominic Zeng Wang and Ingmar Posner. Voting for voting in online point cloud object detection. RSS, 1317, 2015,
[56] Yue Wang, Alireza Fathi, Abbijit Kundu, David Ross, Caroline Pantofaru, Thomas Funkhouser, and Justin Solomon. Pillar-based object detection for autonomous driving. In ECCV, 2020.
[57] Danfei Xu, Dragomir Anguelow, and Ashesh Jain. Pointfusion: Deep sensor fusion for 3d bounding box estimation. In CVPr, pages 243-253, 2018
[58] Yan Yan, Yuxing Mao, and Bo Li. Sccond: Sparsely embedded convolutional detection. Sensors, 18(10)33337, 2018.
[59] Bin Yang, Ming Liang, and Raquel Urtasun. Hdnet: Exploiting hd maps for 3d object detection. In Conference on Robot Learning, pages 155. PMLR, 2018
12

---

[60] Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Real-time 3d object detection from point clouds. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pages 7652-7660, 2018.
[61] Z. Yang. Y. Sun, S. Liu, X. Shen, and J. Jia. Std: Sparse-to-dense 3d object detector for point cloud. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages
2019.
[62] Zetong Yang. Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11040- 1044, 2020.
[63] Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Ipod: Intensive point- object detector for point cloud, 2018.
[64] M. Ye, S. Xu, and T. Cao. Hvnet: Hybrid voxel network for lidar based 3d ohject detection. In 2020 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1628-1637, 2020.
[65] J. Yin, J. Shen, C. Guan, D. Zhou, and R. Yang. Lidar-based online 3d video object detection with graph-based message passing and spatiotemporal transformer attention. In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 11492-11501, 2020.
[66] Ye Yuan and Kris Kitani. Ego-pose estimation and forecasting real-time pd control. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10082-10092 2019.
[67] Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, and Vijay Vasudevan. End-to-end multi-view fusion for 3d object detection in lidar point clouds. In Conference on Robot Learning, pages 923-933 2020.
[68] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR, 2018
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. for practical trade-offs in using the expanded box as the ego-agent's shape and using aggregated points as ground truth to remove the need of human labeling. We also showed the limitation of the StarPoly model in Sec. 5.2 Results.
(c) Did you discuss any potential negative societal impacts of your work? [No] We believe our work will benefit society by providing another perspective to evaluate detection and help set higher safety standard for robotics and autonomous driving.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
2. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] The code and the data are proprietary. We plan to release the code of computing the metrics upon acceptance.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec.3, Sec.4 and Sec.5.
(c) Did you report error bars (e-g., with respect to the random seed after running exper- iments multiple times)? [Yes] Most of our experiments are deterministic. We report variance the proposed StarPoly in the supplementary material.
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We include this information in the supplementary material our proposed StarPoly model.
3. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] We have followed the terms for using the Waymo Open Dataset for pure academic research purposes.
(c) Did you include any new assets either in the supplemental material or URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A] We have followed the terms for using the Waymo Open Dataset for pure academic research purposes.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] We only use the Lidar point clouds and bounding box annotations which do not relate to personally identifiable information or offensive content.
14


Reviews:
[{"review_id": "k6FklBfmI-n", "reviewer": "Reviewer_hKpm", "summary": "This paper presents support distance error (SDE), a new evaluation metric 3D object detection from an egocentric point cloud. In contrast to Intersection over Union (IoU), SDE takes into account the potential collision induced by the object detector’s error over the planned trajectory of an ego-agent. It measures support distance (lateral and longitudinal directions) given the ego-agent’s orientation at given time. This metric is strongly indicative of collision risk.  Furthermore, the paper proposes StarPoly, a new object boundary representation that achieves better performance than traditional bounding box prediction on their new metrics SDE through experiments on autonomous driving Waymo Open dataset.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The ideas of SDE and StarPoly are simple yet more effective in collision measures than standard metric IoU on cuboids. This enables a proper selection of the object detection methods for downstream tasks, such as motion planning, collision avoidance, etc.\n\n(+ Presentation) The paper is well-written in general. Schematic figures (1, 2, 5) are useful for understanding the concepts.\n\n(+ Novel evaluation metric) The SDE-AP and SDE-APD metrics are derived from the longitudinal and lateral distances between the ego-agent and the object. Specifically, it depends on the boundary of the object and the planned trajectory of the ego-agent. The authors have shown in Section 3 that SDE has stronger correlation with the collision than the IoU.\n\n(+ Novel boundary prediction) Due to the strong dependency on the object boundary of the SDE metric, the paper proposes to refine the standard cuboid shape bounding box using the StarPoly model. It takes as input a normalized point cloud extracted from the predicted bounding box and refines it such that it tightly covers the point cloud. With the StarPoly model, the paper shows a significant improvement in their new metric SDE (Section 5), allowing an ego-agent to better measure the potential collision.\n\n(- Generalization) Unlike IoU, SDE is a metric scale metric. It is great for range measurements such as LiDar. However, in many computer vision applications, where physical camera calibration is not available, e.g., single view depth, this metric is not applicable. This is not a critical issue but it is worth highlighting or discussing the limitation of the metric and possible extension of SDE for such cases. And how does this can generalize to 3D SDE?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper mainly focuses on driving dataset, while the concepts can be applied to other types of datasets such as indoor object detection [1].\n\n[1] K. Lai, L. Bo, X. Ren, and D. Fox, “A large-scale hierarchical multi-view rgb-d object dataset,” in Robotics and Automation (ICRA), 2011.\n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "jJIe3OrpJPx", "reviewer": "Reviewer_NNhf", "summary": "This paper presents a new egocentric measure for 3D object detection. The metric better captures the (local) object geometry and bettehr reflect the impact of detection quality on the ego-agent’s safety compared to IoU.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I'm currently on the fence (ie, borderline). I'd be happy to adjust my score provided my concerns are addressed.\n\n---\n\n**Strengths**\n+ The paper is well-written and easy to follow\n+ Literature review is comprehensive\n+ Propose a new metric for 3D object detection with empirical analyses\n+ Present a plug-n-play module that can be directly applied to existing detectors and boost their performance on the new metric\n\n**Concerns**\n- While the SDE metric may better reflect the possibility of collision, it is unclear whether the proposed metric/method is actually necessary/beneficial to the AV stack. In the main paper, the authors motivate SDE and hence the StarPoly merely through the collision of bbox. In practice, instead of directly treating the bbox as occupancy and plan a trajectory accordingly, state-of-the-art planners tend to exploit the perception results as guidance to estimate the cost map and then produce a trajectory. To be more specific, the bbox may be used to extract features, provide guidance on which part the model should attend to, or serve as regularization (through multi-task learning), but usually will not be used for explicit collision modeling. It would be thus more convincing if the proposed method is hooked up and evaluated with downstream planning/navigation tasks. Otherwise, even if a detector has better SDE-AP(D) (as shown in Sec. 5), we are still not sure if it really helps on navigation (which is the hope of such ego-centric metric).\n\n- SDE, in some sense, captures the local geometry of the object (the part where it's used to compute the distance). Ideally the lower the SDE, the closer the shape is to the GT shape, the more consistent the collision estimation would be. To verify this, it would be great if the authors re-compute the numbers for Tab. 1, but this time using StarPoly to determine collision. Ideally the gap should be as obvious as using bbox.\n\n- SDE is the max of latitude and longitudinal distance. Which one plays more important role? Intuitively I would guess latitude is more critical. Would be great if the authors could provide more analyses and insights.\n\n- Why isn't  StarPoly in Tab. 2 GT column? Is StarPoly also inferior when using GT bbox?\n\n- The output of StarPoly looks like a tighter rectangle with smooth corners and edges. is it really necessary to adopt a network to perform the task? how about just compute the mean shape/contour for all the vehicles, then during inference simply adjust the scale and directly apply? how does it work?\n\n- The current CVC baseline is too weak. Seems a bit unfair to me. Vehicles has a very strong geometry prior - symmetry, and one should leverage it. For instance, use the longitudinal axis to flip (double) the point clouds and then compute CVC. This would provide a much better shape geometry. Furthermore, based on Fig. 10, the numbers of CVC looks pretty good, even better than StarPoly (3 out of 4 times). Is this a coincidence? If yes, would be great if the authors show the cases where CVC fails. Also, would be great if the authors could show which part of the geometry/contour is used to compute SDE in the images so that the reader can better understand what results in the error (something similar to Fig. 4).\n\n- Why not simply use the points within the predicted bbox to represent the shape, estimate collision, and compute SDE metrics? Bbox are a coarse representation of the object geometry, while the surface points are more fine-grained. Can the authors comment about this?\n\n- The authors should demonstrate the effectiveness with a real planner and showcase how the SDE based metrics transform to the downstream task comparing to IoU based metrics. \n\n- L232: While the end results is the same, the explanation is a bit flawed. The barycenter loss is in fact encouraging x to lie on the same side as h wrt to the line formed by $v_r$ and $v_l$. It isn't forcing x to be in the triangle. \n\n**Minor suggestions**\n- I had a hard time understanding why we need SDE@t: since the authors adopt the GT rigid transform to warp the detection box, they are essentially assuming that the prediction error will remain the same (wrt the the GT box) throughout the time. It doesn't make much sense to me, why is this useful? Would be great if the authors could provide an example.\n\n- The way StarPoly is parameterized restricted it to only handle non-convex shapes; the method is very similar to active contour methods (ie snake). The authors might want to cite a few papers there.\n\n- I'm very confused how exactly is $SD_\\alpha(B(o), e)$ computed at first. Based on Eq. 1 and the surrounding text (L137-L141), it is the minimum point-to-line distance among all the points *on the (GT) object boundary* wrt the lateral/longitudinal line. It is not clear what is the object boundary and what those points exactly are. It may be the object bbox and the points are sampled from the box boundary; or it may be the real object shape represented by LiDAR points enclosed by the bbox. In implementation details (L158-L161), the authors point out that aggregated LiDAR points are used to represent the (GT) shape. I thus assume its the latter for both. However, the illustrations in Fig. 1 and Fig. 4 suggest otherwise. Things are clearer when I read Sec. 4.2. Would be great if the authors make this more explicit. (I assume for CVC and StarPoly the points are also sampled from the boundary?)\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors include the limitations of their model.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "XRsLcDYCbyd", "reviewer": "Reviewer_irMp", "summary": "This paper proposes a novel ego-centric evaluation criteria for 3D object detection. The proposed metric, SDE, measures error in 3D object detection prediction in terms of lateral and longitudinal distances w.r.t. the ego vehicle. Moreover, the paper proposes a novel approach to represent objects as predict amodal contours, which is used to improve the proposed SDE metric on the Waymo Open dataset.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Positives**  \nP1. The paper proposes a metric which incorporates two properties: (a) evaluating 3D object detection prediction w.r.t. the ego vehicle, which is directly correlated to the chances of collision w.r.t. the ego vehicle. (b) defining lateral and longitudinal distance supports to capture the parts of the boundary which are closer to the ego-agent.   \nP2. The paper proposes a new optimization approach which predicts more tightly fitted amodal contour outlines of the detected LIDAR bounding boxes.   \nP3. The paper analyzes the usefulness of the metric and the StarPoly approach with their metric on the Waymo Open dataset.\n\n**Negatives**  \nN1. While the proposed SDE metric is novel in two aspects (P1a and P1b), it would be great if the paper analyzes the metrics by breaking them down one by one. This can be shown in Table 1 and Fig. 2, while comparing the proposed SDE metric w.r.t. IoU. While IoU is not an ego-centric measure, it would be interesting to define an ego-centric version of IoU measure which can help us determine how well a 3D bounding box prediction w.r.t. ego vehicle correlates with CDA? This would break down the importance of defining an ego-centric measure as well as focusing on the parts of the boundary which are more important from the perspective of the ego vehicle.  \nN2. Should there be a normalization factor for the definition of $ SD_{\\alpha} $ (Eq. 2), especially when $ \\alpha = lon $? Larger distances to these support distance measures mean that the difference w.r.t. GT would be higher (Eq. 2). This would impact definition of SDE, as the minimum of its lateral and longitudinal measures. Hence it is not clear whether those two support distance measures are on the same scale or not.  \nN3. It is not clear what is the purpose of proposing SDE@t metric? Besides a couple of plots to motivate certain aspects of the paper, it is not clear what is the usefulness of this measure and the understanding we gain from this? \n\n\n**Questions**    \nSome things weren't clear.  \nQ1. In Fig. 3, what is CDA? What does mean correspond to here? Is it mean across all the object instances?  \nQ2. In l.164-165, it is mentioned that SDE_{lat} is very useful indicator to reflect collision risk. But isn't SDE_{lon} also a useful indicator, especially for head-on collisions?  \nQ3. In l.137, while defining \"longitudinal line\", shouldn't the line be passing through the center as well?  \nQ4. Why is GT box column (col 5) in Table 2 not 0? Doesn't Eq 2 collapse to 0 here?  \nQ5. While comparing with [33] in supplementary material, it is mentioned that one of the benefits of this approach is stability. It is not clear how this is an advantage. Since the proposed ego-centric measure focuses on correlation with the other objects, such 3D bounding box detections will always be tied with a planner. While [33] uses a particular predicted planner, this approach relies on the ground truth rigid body motion. So essentially this approach also relies on a particular planner algorithm. It is not clear why this approach which propagates errors in its planner algorithm better than [33]. In terms of stability, while measuring one detection algorithm to another, the planner component can always be kept constant. This point is not quite clear.   \nQ6. In the analysis of the SDE metric, it would be interesting to see how the human annotation of the bounding boxes \"agree\" with the SDE metric?  \nQ7. Does the StarPoly approach gives better bounding box predictions in terms of the IoU metric?  \nQ8. It is mentioned in l.208 that the bounding box is expanded for input to the StarPoly PointNet model. It would be good to have an ablation study which can show how this factors into the proposed algorithm. If the StarPoly algorithm is not very sensitive to this, wouldn't different algorithms with different detector behavior can have similar outcomes with the StarPoly algorithm? If the StarPoly algorithm is indeed sensitive to this, it would be interesting if the authors talked about how this algorithm is sensitive to the expansion and also the different detection algorithms. While, the latter is shown to some extent in the paper, but it would be useful to analyze (quantitatively and qualitatively) the corresponding StarPoly predictions from the different behaviors of the different detector algorithm,", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This has been talked about to some extent. ", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5-6"}, "extra_text": {}}, {"review_id": "SYtppvcD4OB", "reviewer": "Reviewer_D4yU", "summary": "The paper proposed 1) a new metric SDE measuring the error in predicted lateral and longitude distance from the ego-car to another car, 2) an object contour prediction model to refine the shape from a simple box. 3) SDE-based AP and weighted AP for evaluating object detectors.\nThe proposed SDE metric better reflects the risk of collision considering the ego-car trajectory.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality/Significance\n1) The idea of SDE in lateral and longitude distances is relatively novel, making sense, and might be very practical for the purpose of collision prevention. However, as the authors mentioned that the sign of SDE has different meanings, I think we should keep the sign and use different thresholds for SDE-based AP. Note that it is riskier for positive SDE but less efficient for negative SDE.\nMoreover, the examples in Fig. 4 (c,d) both have negative SDEs which means these tight boxes are only more conservative than a contour. Hence, I do not see a strong reason to compare different bounding box-based detectors using SDE-based AP. As shown in Fig. 7, the ranking of detectors is mostly unchanged (except [0m, 5m]). Moreover, we cannot see AP with respect to positive or negative SDE.\n2) The proposed StarPoly is simple (limited novelty) and making sense considering SDE is computed using group truth aggregated point clouds. It is clear that StarPoly will increase SDE-based AP comparing to a box-based detector, but how critical is the improvement with respect to planning the ego agent to avoid a collision? I think an additional evaluation is needed to convince me.\n\nTo Summarized, SDE is an interesting new concept but without considering the sign it is limited. I am also not sure how SDE can be used to select a different box-based detector. Finally, do we really need to estimate the contour to avoid collision in most cases? Given the current evaluation, I am not convinced.\n\nQuality/Clarity\nThe paper is easy to read with good illustrating figures.\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No strong limitations and negative social impacts.", "review_text": "", "extra_scores": {"ethical_concerns": "No strong ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the rebuttal most reviewers agreed to accept this paper. The remaining concerns centered around connections of the proposed evaluation method and algorithm to down-stream planning and driving performance. The AC agrees that the paper would be stronger making these connections, but sees enough merit in the proposed work to accept the paper without it.
Decision: Accept (Poster)