Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a novel ego-centric evaluation criteria for 3D object detection. The proposed metric, SDE, measures error in 3D object detection prediction in terms of lateral and longitudinal distances w.r.t. the ego vehicle. Moreover, the paper proposes a novel approach to represent objects as predict amodal contours, which is used to improve the proposed SDE metric on the Waymo Open dataset.
main_review: **Positives**  
P1. The paper proposes a metric which incorporates two properties: (a) evaluating 3D object detection prediction w.r.t. the ego vehicle, which is directly correlated to the chances of collision w.r.t. the ego vehicle. (b) defining lateral and longitudinal distance supports to capture the parts of the boundary which are closer to the ego-agent.   
P2. The paper proposes a new optimization approach which predicts more tightly fitted amodal contour outlines of the detected LIDAR bounding boxes.   
P3. The paper analyzes the usefulness of the metric and the StarPoly approach with their metric on the Waymo Open dataset.

**Negatives**  
N1. While the proposed SDE metric is novel in two aspects (P1a and P1b), it would be great if the paper analyzes the metrics by breaking them down one by one. This can be shown in Table 1 and Fig. 2, while comparing the proposed SDE metric w.r.t. IoU. While IoU is not an ego-centric measure, it would be interesting to define an ego-centric version of IoU measure which can help us determine how well a 3D bounding box prediction w.r.t. ego vehicle correlates with CDA? This would break down the importance of defining an ego-centric measure as well as focusing on the parts of the boundary which are more important from the perspective of the ego vehicle.  
N2. Should there be a normalization factor for the definition of $ SD_{\alpha} $ (Eq. 2), especially when $ \alpha = lon $? Larger distances to these support distance measures mean that the difference w.r.t. GT would be higher (Eq. 2). This would impact definition of SDE, as the minimum of its lateral and longitudinal measures. Hence it is not clear whether those two support distance measures are on the same scale or not.  
N3. It is not clear what is the purpose of proposing SDE@t metric? Besides a couple of plots to motivate certain aspects of the paper, it is not clear what is the usefulness of this measure and the understanding we gain from this? 


**Questions**    
Some things weren't clear.  
Q1. In Fig. 3, what is CDA? What does mean correspond to here? Is it mean across all the object instances?  
Q2. In l.164-165, it is mentioned that SDE_{lat} is very useful indicator to reflect collision risk. But isn't SDE_{lon} also a useful indicator, especially for head-on collisions?  
Q3. In l.137, while defining "longitudinal line", shouldn't the line be passing through the center as well?  
Q4. Why is GT box column (col 5) in Table 2 not 0? Doesn't Eq 2 collapse to 0 here?  
Q5. While comparing with [33] in supplementary material, it is mentioned that one of the benefits of this approach is stability. It is not clear how this is an advantage. Since the proposed ego-centric measure focuses on correlation with the other objects, such 3D bounding box detections will always be tied with a planner. While [33] uses a particular predicted planner, this approach relies on the ground truth rigid body motion. So essentially this approach also relies on a particular planner algorithm. It is not clear why this approach which propagates errors in its planner algorithm better than [33]. In terms of stability, while measuring one detection algorithm to another, the planner component can always be kept constant. This point is not quite clear.   
Q6. In the analysis of the SDE metric, it would be interesting to see how the human annotation of the bounding boxes "agree" with the SDE metric?  
Q7. Does the StarPoly approach gives better bounding box predictions in terms of the IoU metric?  
Q8. It is mentioned in l.208 that the bounding box is expanded for input to the StarPoly PointNet model. It would be good to have an ablation study which can show how this factors into the proposed algorithm. If the StarPoly algorithm is not very sensitive to this, wouldn't different algorithms with different detector behavior can have similar outcomes with the StarPoly algorithm? If the StarPoly algorithm is indeed sensitive to this, it would be interesting if the authors talked about how this algorithm is sensitive to the expansion and also the different detection algorithms. While, the latter is shown to some extent in the paper, but it would be useful to analyze (quantitatively and qualitatively) the corresponding StarPoly predictions from the different behaviors of the different detector algorithm,
limitations_and_societal_impact: This has been talked about to some extent.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a new egocentric measure for 3D object detection. The metric better captures the (local) object geometry and bettehr reflect the impact of detection quality on the ego-agent’s safety compared to IoU.
main_review: I'm currently on the fence (ie, borderline). I'd be happy to adjust my score provided my concerns are addressed.

---

**Strengths**
+ The paper is well-written and easy to follow
+ Literature review is comprehensive
+ Propose a new metric for 3D object detection with empirical analyses
+ Present a plug-n-play module that can be directly applied to existing detectors and boost their performance on the new metric

**Concerns**
- While the SDE metric may better reflect the possibility of collision, it is unclear whether the proposed metric/method is actually necessary/beneficial to the AV stack. In the main paper, the authors motivate SDE and hence the StarPoly merely through the collision of bbox. In practice, instead of directly treating the bbox as occupancy and plan a trajectory accordingly, state-of-the-art planners tend to exploit the perception results as guidance to estimate the cost map and then produce a trajectory. To be more specific, the bbox may be used to extract features, provide guidance on which part the model should attend to, or serve as regularization (through multi-task learning), but usually will not be used for explicit collision modeling. It would be thus more convincing if the proposed method is hooked up and evaluated with downstream planning/navigation tasks. Otherwise, even if a detector has better SDE-AP(D) (as shown in Sec. 5), we are still not sure if it really helps on navigation (which is the hope of such ego-centric metric).

- SDE, in some sense, captures the local geometry of the object (the part where it's used to compute the distance). Ideally the lower the SDE, the closer the shape is to the GT shape, the more consistent the collision estimation would be. To verify this, it would be great if the authors re-compute the numbers for Tab. 1, but this time using StarPoly to determine collision. Ideally the gap should be as obvious as using bbox.

- SDE is the max of latitude and longitudinal distance. Which one plays more important role? Intuitively I would guess latitude is more critical. Would be great if the authors could provide more analyses and insights.

- Why isn't  StarPoly in Tab. 2 GT column? Is StarPoly also inferior when using GT bbox?

- The output of StarPoly looks like a tighter rectangle with smooth corners and edges. is it really necessary to adopt a network to perform the task? how about just compute the mean shape/contour for all the vehicles, then during inference simply adjust the scale and directly apply? how does it work?

- The current CVC baseline is too weak. Seems a bit unfair to me. Vehicles has a very strong geometry prior - symmetry, and one should leverage it. For instance, use the longitudinal axis to flip (double) the point clouds and then compute CVC. This would provide a much better shape geometry. Furthermore, based on Fig. 10, the numbers of CVC looks pretty good, even better than StarPoly (3 out of 4 times). Is this a coincidence? If yes, would be great if the authors show the cases where CVC fails. Also, would be great if the authors could show which part of the geometry/contour is used to compute SDE in the images so that the reader can better understand what results in the error (something similar to Fig. 4).

- Why not simply use the points within the predicted bbox to represent the shape, estimate collision, and compute SDE metrics? Bbox are a coarse representation of the object geometry, while the surface points are more fine-grained. Can the authors comment about this?

- The authors should demonstrate the effectiveness with a real planner and showcase how the SDE based metrics transform to the downstream task comparing to IoU based metrics. 

- L232: While the end results is the same, the explanation is a bit flawed. The barycenter loss is in fact encouraging x to lie on the same side as h wrt to the line formed by $v_r$ and $v_l$. It isn't forcing x to be in the triangle. 

**Minor suggestions**
- I had a hard time understanding why we need SDE@t: since the authors adopt the GT rigid transform to warp the detection box, they are essentially assuming that the prediction error will remain the same (wrt the the GT box) throughout the time. It doesn't make much sense to me, why is this useful? Would be great if the authors could provide an example.

- The way StarPoly is parameterized restricted it to only handle non-convex shapes; the method is very similar to active contour methods (ie snake). The authors might want to cite a few papers there.

- I'm very confused how exactly is $SD_\alpha(B(o), e)$ computed at first. Based on Eq. 1 and the surrounding text (L137-L141), it is the minimum point-to-line distance among all the points *on the (GT) object boundary* wrt the lateral/longitudinal line. It is not clear what is the object boundary and what those points exactly are. It may be the object bbox and the points are sampled from the box boundary; or it may be the real object shape represented by LiDAR points enclosed by the bbox. In implementation details (L158-L161), the authors point out that aggregated LiDAR points are used to represent the (GT) shape. I thus assume its the latter for both. However, the illustrations in Fig. 1 and Fig. 4 suggest otherwise. Things are clearer when I read Sec. 4.2. Would be great if the authors make this more explicit. (I assume for CVC and StarPoly the points are also sampled from the boundary?)


limitations_and_societal_impact: The authors include the limitations of their model.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposed 1) a new metric SDE measuring the error in predicted lateral and longitude distance from the ego-car to another car, 2) an object contour prediction model to refine the shape from a simple box. 3) SDE-based AP and weighted AP for evaluating object detectors.
The proposed SDE metric better reflects the risk of collision considering the ego-car trajectory.
main_review: Originality/Significance
1) The idea of SDE in lateral and longitude distances is relatively novel, making sense, and might be very practical for the purpose of collision prevention. However, as the authors mentioned that the sign of SDE has different meanings, I think we should keep the sign and use different thresholds for SDE-based AP. Note that it is riskier for positive SDE but less efficient for negative SDE.
Moreover, the examples in Fig. 4 (c,d) both have negative SDEs which means these tight boxes are only more conservative than a contour. Hence, I do not see a strong reason to compare different bounding box-based detectors using SDE-based AP. As shown in Fig. 7, the ranking of detectors is mostly unchanged (except [0m, 5m]). Moreover, we cannot see AP with respect to positive or negative SDE.
2) The proposed StarPoly is simple (limited novelty) and making sense considering SDE is computed using group truth aggregated point clouds. It is clear that StarPoly will increase SDE-based AP comparing to a box-based detector, but how critical is the improvement with respect to planning the ego agent to avoid a collision? I think an additional evaluation is needed to convince me.

To Summarized, SDE is an interesting new concept but without considering the sign it is limited. I am also not sure how SDE can be used to select a different box-based detector. Finally, do we really need to estimate the contour to avoid collision in most cases? Given the current evaluation, I am not convinced.

Quality/Clarity
The paper is easy to read with good illustrating figures.





limitations_and_societal_impact: No strong limitations and negative social impacts.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents support distance error (SDE), a new evaluation metric 3D object detection from an egocentric point cloud. In contrast to Intersection over Union (IoU), SDE takes into account the potential collision induced by the object detector’s error over the planned trajectory of an ego-agent. It measures support distance (lateral and longitudinal directions) given the ego-agent’s orientation at given time. This metric is strongly indicative of collision risk.  Furthermore, the paper proposes StarPoly, a new object boundary representation that achieves better performance than traditional bounding box prediction on their new metrics SDE through experiments on autonomous driving Waymo Open dataset.
main_review: The ideas of SDE and StarPoly are simple yet more effective in collision measures than standard metric IoU on cuboids. This enables a proper selection of the object detection methods for downstream tasks, such as motion planning, collision avoidance, etc.

(+ Presentation) The paper is well-written in general. Schematic figures (1, 2, 5) are useful for understanding the concepts.

(+ Novel evaluation metric) The SDE-AP and SDE-APD metrics are derived from the longitudinal and lateral distances between the ego-agent and the object. Specifically, it depends on the boundary of the object and the planned trajectory of the ego-agent. The authors have shown in Section 3 that SDE has stronger correlation with the collision than the IoU.

(+ Novel boundary prediction) Due to the strong dependency on the object boundary of the SDE metric, the paper proposes to refine the standard cuboid shape bounding box using the StarPoly model. It takes as input a normalized point cloud extracted from the predicted bounding box and refines it such that it tightly covers the point cloud. With the StarPoly model, the paper shows a significant improvement in their new metric SDE (Section 5), allowing an ego-agent to better measure the potential collision.

(- Generalization) Unlike IoU, SDE is a metric scale metric. It is great for range measurements such as LiDar. However, in many computer vision applications, where physical camera calibration is not available, e.g., single view depth, this metric is not applicable. This is not a critical issue but it is worth highlighting or discussing the limitation of the metric and possible extension of SDE for such cases. And how does this can generalize to 3D SDE?

limitations_and_societal_impact: The paper mainly focuses on driving dataset, while the concepts can be applied to other types of datasets such as indoor object detection [1].

[1] K. Lai, L. Bo, X. Ren, and D. Fox, “A large-scale hierarchical multi-view rgb-d object dataset,” in Robotics and Automation (ICRA), 2011.
review_title: null

Meta-review: After the rebuttal most reviewers agreed to accept this paper. The remaining concerns centered around connections of the proposed evaluation method and algorithm to down-stream planning and driving performance. The AC agrees that the paper would be stronger making these connections, but sees enough merit in the proposed work to accept the paper without it.
Decision: Accept (Poster)