Towards Long-Tailed 3D Detection
Authors: Neehar Peri
Achal Dave
Deva Ramanan
Shu Kong
Venue: CoRL 2022 Poster (2022)
Keywords: Autonomous Vehicles
Long-Tailed 3D Detection
Multimodal Fusion

Abstract: Contemporary autonomous vehicle (AV) benchmarks have advanced techniques for training 3D detectors, particularly on large-scale lidar data. Surprisingly, although semantic class labels naturally follow a long-tailed distribution, contemporary benchmarks focus on only a few common classes (e.g., pedestrian and car) and neglect many rare classes in-the-tail (e.g., debris and stroller).
However, AVs must still detect rare classes to ensure safe operation. Moreover, semantic classes are often organized within a hierarchy, e.g., tail classes such as child and construction-worker are arguably subclasses of pedestrian. However, such hierarchical relationships are often ignored, which may lead to misleading estimates of performance and missed opportunities for algorithmic innovation. We address these challenges by formally studying the problem of Long-Tailed 3D Detection (LT3D), which evaluates on all classes, including those in-the-tail. We evaluate and innovate upon popular 3D detection codebases, such as CenterPoint and PointPillars, adapting them for LT3D.
We develop hierarchical losses that promote feature sharing across common-vs-rare classes, as well as improved detection metrics that award partial credit to "reasonable" mistakes respecting the hierarchy (e.g., mistaking a child for an adult). Finally, we point out that fine-grained tail class accuracy is particularly improved via multimodal fusion of RGB images with LiDAR; simply put, small fine-grained classes are challenging to identify from sparse (lidar) geometry alone, suggesting that multimodal cues are crucial to long-tailed 3D detection. Our modifications improve accuracy by 5% AP on average for all classes, and dramatically improve AP for rare classes (e.g., stroller AP improves from 3.6 to 31.6).

Towards Long-Tailed 3D Detection
Nechar Peri', Achal Dave', Deva Ramanan 122 Shu Kong" 'Robotics Institute, Carnegie Mellon University Argoo AI, DDptt. of Computer Science and Engineering, Texas A&M University {mperi, adave, deva) } andrew.. cmu. emu.edu shu@tamu. edu
Abstract: Contemporary autonomous vehicle (AV) benchmarks have advanced techniques for training 3D detectors, particularly on large-scale LiDAR data. Sur- prisingly, although semantic class labels naturally follow a long-tailed distribution, these benchmarks focus on only few common classes (c.g., pedestrian and car) and negiect many rare classes in-the-tail (c.g., debris and stroller). However, in the real open world, AVs must still detect rare classes to ensure safe operation. Moreover, semantic classes are often organized within a hierar- chy, e.g.. tail classes such as child ih and construction subclasses of pedestriam However, such hierarchical relationships are often ignored, which may yield misleading estimates of performance and missed op- portunities for algorithmic innovation. We address these challenges by formally studying the problem of Long-Tailed 3D Detection (LT3DD) which evaluates on all classes, including those in-the-tail. We evaluate and innovate upon popular 3D detectors, such as CenterPoint and PointPillars, adapting them for LT3D. We develop hierarchical losses that promote feature sharing across common-vs-rare classes, as well as improved detection metrics that award partial credit to "reason- able" mistakes respecting the hierarchy (e.g., mistaking chils 1 for adult).. Finally, we point out that fine tnee-graied tail class accuracy is particularly improved via multimodal fusion of RGB images with LiDAR; simply put, fine-grained classes are challenging to identify from sparse (LiDAR) geometry alone, suggesting that multimodal cues crucial to long-tailed 3D detection. Our modifications improve accuracy by 5% AP on average for all classes, and dramatically improve AP for rare classes (e.g., stroller AP improves from 0.1 to 31.6)!
are arguably
Keywords: Autonomous Vehicles, Long-Tailed 3D Detection, Multimodal Fusion
1
Introduction
3D object detection is key component in many robotics systems such as autonomous vehicles (AVs) [1, 2]. To facilitate research in this space, the AV industry has released large-scale 3D annotated multimodal datasets [2, 3, 4]. However, these datasets benchmark on only a few common classes such as pedestrian and car. r. In the real open world, safe navigation [5, 6] requires AVs to reliably detect rare-class objects such as chi 1d and sorleer. This motivates Long-Tailed 3D Detection (LT3D), problem requiring detecting objects from both common and rare classes.
Status Quo. Among contemporary AV datasets, nuScenes [2] has exhaustively annotated objects of various classes crucial to AVs (Fig. 1) and organized them with semantic hierarchy (Fig. 3). As focuses on only a few (common) classes, prior works miss opportunities exploit this semantic hierarchy during training. We argue that these benchmarking protocols are flawed because detecting fine-grained classes is useful for downstream tasks such as motion planning. This motivates us to study LT3D by re-purposing all annotated classes in nuScenes.
Protocol. LT3D requires 3D localization and recognition of objects from each of the common (e.g., adult and car) and rare classes (e.g, child lhild and stroller) Moreover, for safety-critical robots such as autonomous vehicles, believe detecting but mis-classifying rare objects (e.g.. mis-classifyingg a chile 1 as an adult) t) is preferable to failing to detect them at all. Therefore, we propose a new metric to quantify the severity of classification mistakes LT3D that exploits inter-class relationships to award partial credit (Fig. 3). We use both the standard and proposed metrics to evaluate 3D detectors on all classes.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Few
Medi um
Many
Personal Mobilyy
Emergency Vah dle
Police Officer
Puahesll e-pu 1ab Saperdles
Dubris
Const Construction Worker
Motoreycle
Construction Vahide
Common Ofjects
Bus
Pushable Pullable
Trailer
Barner
Adult
Car
107
107
Objects
Count
Figure According to the histogram of per-class object counts (on the left), the nuScenes benchmark fo- cuses on the common classes in cyan (e.g., car and barrier) but ignores rare ones in red (e.g.. stroller and s)b In fact. the benchmark creates superclass pedestrian by grouping multiple classes in green, including the common class adult and several rare classes (c-g., child d and pol colice-on co-of oofficer)) cer); this complicates the analysis detection performance as pedest sindestrian ian performance is dominated by adult. More- over, the ignored superclass pushable e-pulllllll able also contains diverse objects such shoppi Shopping-Con pappinggcrt,, dolly, luggage and trash-car as shown in the top row (on the right). We argue that AVs should also detect classes they affect AV behaviors. Following [7], report performance for three groups of classes based on their cardinality (split by dotted lines): Many, Medi um, and Few.
Technical Insights. To address LT3D, we first retrain state-of-the-art LiDAR-based 3D detectors on all classes. Naively retraining detectors produces poor performance on rare classes (e.g.. yielding 0.1 AP on child and 0.1 AP on stroller). We propose several algorithmic innovations to improve these results. First, encourage feature sharing across common-vas v-rrre classes, we learn single feature trunk, adding in hierarchical coarse classes that ensure features will be useful for both common and rare classes. Second, we find that LiDAR data is simply impoverished for even humans to recognize certain tail objects that tend to be small, such as strollers, We explore multimodal fusion, and introduce simple approach that post-processes single-modal 3D detections from LiDAR and RGB inputs, filtering away detections that are inconsistent across modalities. Our innovations significantly improve performance on LT3D by 5 % AP on average, greatly boosting performance when allowing for partial credit (e.g., achieving 16.9 38.8 AP for child stroller).
Contributions. We make three major contributions. First, we formulate the problem LT3D, empha- sizing detection both common and rare classes safety-critical AVs. Second, we design LT3D's benchmarking protocol and develop a supplemental metric that awards partial credit depending on the severity of misclassifications (c.g., misclassifying child-vs-adull is less problematic than misclassifying chi ld-vs-car). Third, we propose several LT3D, including simple multimodal fusion technique that uses RGB- based detections filter out false- possivvositive LiDAR based detections, leading to significant improvement of ra rare-clas detection.
approaches to
Related Works
3D Object Detection. Contemporary benchmarks, often the AV setting, favor LiDAR-baase detectors, emphasizing common classes and ignoring rare ones. Approaches to 3D detection usually adopt an anchor-based model architecture that defines per-class shapes to guide class-aware object detection [8, 10, 11, 12]. recent anchor-free model, CenterPoint [13] achieves the state- of-the-art art for LiDAR- based 3D object detection. Specifically, it learns to predict an object's 's center and estimates the 3D shape for each detected object's center. Existing LiDAR-baa
3D detectors exclusively focus on data from common classes [8, 9, 13] and do not study how to detect rare classes. RGB-based 3D detectors underperform LiDAR- -based methods because the monocular RGB input does not provide reliable 3D measures (unlike LiDAR). As result, RGB-based 3D detectors are not widely adopted. However, exploring LT3D we find that RGB-detectors shine for detecting objects of rare classes. Importantly, multimodal fusion significantly improves LT3D.

---

Multimodal 3D Detection. Conventional wisdom suggests that fusing multimodal cues, particularly using LiDAR and RGB, can improve 3D detection. Intuitively, LiDAR faithfully measures the 3D world (although it has notoriously sparse point returns), and RGB is high-resolution (but lacks 31> information). Multimodal fusion for 3D detection is an active field of exploration. Existing methods suggest different ways to fuse the two modalities. Proposed methods encode separate modalities and fuse object proposals [14, 15, 16, 17. 18, 19), augment LiDAR points with either RGB features [20], augment RGB images with LiDAR points [21] or add semantic information obtained by processing RGB inputs 122, 23]. Others propose stage-wise methods that first detect boxes over images and localize in 3D with LiDAR [24] and fuse detections from single-modas detectors [25, 26]. While the above methods have not been tested for LT3D, our work shows that RGB is a key modality for LT3D.
Loig-Tailed Perception (LTP). Real-world data tends to follow long-tailed class distributions [27], i.e. few classes are dominant in the data, while many others are rarely seen. L.TP is long-standing problem in the literature [7]. It has been widely studied through the lens of image classification and requires training on class-imbalanced data, aiming for high accuracy averaged across imbalanced
classes [7, 28. 29J. Existing methods propose reweighting losses [30, 31, 32, 33, 34, 35], rebalancing data sampling [36, 38], balancing gradients computed from imbalanced classes [39], and balancing network weights [29]. Others study LTP through the lens of 2D object detection over RGB images [40]. Compared to 2D image- based recognition, 3D long-taile tailed detection has unique opportunities and challenges because sensors such as LiDAR directly provide geometric and ego-motion cues that are difficult to extract from 2D images. 2D detectors must detect objects of different scales due to perspective image projection, dramatically increasing the complexity of the output space (c.g.. requiring more anchor boxes). In contrast, 31) objects do not exhibit as much scale variation, but far-away objects tend to be sparse, imposing different challenges. Finally, 3D detectors often use class-aware heads (i.e. each class has its own binary classifier) while 2D long-tail recognition typically use shared softmax heads (which may make it easier to enforce hierarchies, as explained above). To the best of our knowledge, long-tailed 3D detection (LT3D) has not yet been explored. In LT3D, we find a unique challenge: rare classes are not only infrequent but are also difficult to distinguish using LiDAR alone. This motivates us to use RGB to complement LiDAR. We find using both RGB (for better recognition) and LiDAR (for better 3D localization) helps detect rare classes.
3 LT3D: Methodology
To approach LT3D, we first retrain SOTA 3D detectors on all classes, including LiDAR-based detectors (PointPillars [8]nand CenterPoint [13]), an RGB-based detector (FCOS3D [41]), and multimodal detector (TransFusion [17]). We further introduce several modifications that consistently improve their LT3D performance. Please refer to the supplement for training details.
Grouping-Free Detector Head. Extending existing 3D detectors to train with more classes is surprisingly challenging. Many contemporary networks use a multi-head architecture that groups classes of similar size and shape to facilitate efficient feature sharing. For example, CenterPoint groups pedest rian and traffic- cone since these objects are both tall and skinny. However, multi-headed grouping strategies may not work for diverse classes like pushable -pulppuli- pullabll and debris and are difficult to scale for a large number of classes. Therefore, we first consider making each class its own group to avoid hand-crafted grouping heuristics. However, the multi-head architecture has per-class detectors that consist of multiple layers with lots of parameters, hence learning them easily overfits to rare-classes. Our final solution is to merge all classes into a single group with a proportionally heavier detector head to simplify training. Our group-free (i.e. single- head) architecture has shared backbone across all classes, and each class has only one linear layer as its detector. This significantly reduces the number of parameters and allows lcarning the shared feature backbone collaboratively with all classes, effectively mitigating overfitting to rare-classes. Adding a new class is as simple as adding a single linear layer to the detector head. Our grouping- free detector head achieves improved accuracy over grouping-base based methods, as shown in the supplement.
Training with Semantic Hierarchies. nuScenes defines a semantic hierarchy (Fig. 3) for all classes, grouping semantically similar classes under coarse- -gainedd categories. We leverage this hierarchy during training. Specifically, we train detectors to predict three labels for each object: its fine-grained label (e.g., chi dd its coarse class (e.g., pedest r rin),, and the root class object. We adopt grouping-free free detector head that outputs separate "multitask" heatmaps for each class, and use a per-class sigmoid focal loss rather than multi-class cross-entropy loss. It is worth noting that this
3

---

LiDAR-base Detections
RGB-based Detections
Filtered LiDAR- based Detections
Multimodal filtering effectively removes high-scoring false- dalse-positive LiDAR detections. The green boxes are ground- truth st stroilers, while the blue boxes are stroiles detections from our best performing models, liDAR-based detector CenterPoint [13] (left) and RGB-based detector FCOS3D [41] (mid). The final filtered result removes LiDAR detections not within m meters of any RGB detection (right).
Figure 2:
simple "multitask" learning strategy does not necessarily enforce hierarchy, hence can extend to more complex label relationships. Crucially, because we do not employ softmax losses, adding a vehicle heatmap does not directly interfere with the car heatmap (as they would with multi-clas, softmax loss). However, this might produce repeated detections on the same test object. We address that by simply ignoring coarse detections at test time. We explore alternatives in the supplement and conclude that they achieve similar LT3D performance. Perhaps surprisingly, this training method improves detection performance not only for rare classes, but also for common classes.
Augmentation Schedule. Class-balanced resampling a common technique in learning with long- tailed classes. This augmentation strategy increases the number of rare objects seen in training but skews the class distribution and leads to more false positives for rare classes inference. Prior works [22, 17] suggest disabling class-balanced resampling for the last few training epochs to better match the real class distribution, reducing false positives. We validate this approach in training 3D detectors and find that it often improves performance for rare classes the cost of common classes.
Multimodal Fusion by Filtering. Small fine mne-grained classes are challenging to identify from sparse (LiDAR) geometry alone, suggesting that multimodal cues can improve long-tailed detection. We evaluate several multimodal fusion algorithms, but find simple strategy of post-hoe filtering to work remarkably well. Although
detectors are widely adopted for 3D detection, we find that they produce many high-scoring false positives (FPs) for rare classes due to misclassification. We focus on removing such FPs. To this end, we use an RGB-based detector to filter out high-scoring false-positive LiDAR detections by leveraging two insights: (1) LiDAR -bsee 3D-detectors are accurate w.r.t 3D localization and yield high recall (though classification is poor), and (2) RGB-based 3D-detections are accurate w.r.t recognition (though 3D localization is poor). Fig. demonstrates this filtering strategy. For each RGB-based detection, we keep LiDAR- based detections within radius of m meters and remove all the others (that are not close to any RGB-based detections). We use FCOS3D [41] as the (GB-based detector in this work.
LT3D: Evaluation Protocol
Conceptually, LT3D extends the traditional 3D detection problem, which focuses on identifying objects from K common classes, by further requiring detection of N rare classes. As LT3D emphasizes detection performance on all classes, we report the metrics for three groups of classes based on their cardinality (Fig. 1-left): many (>50k instances per class), medium (5k~50k), and and (<5k). We describe the metrics below.
Standard Detection Metrics. Mean average precision (mAP) is an established metric for object detection [42, 1, 43]. For 3D detection on LiDAR sweeps, a true positive (TP) is defined as a detection that has center distance within a distance threshold on the ground-plane to ground-truth annotation [2]. mAP computes the mean of AP over classes, where per-class AP is the area under the precision-recall curve, and distance thresholds of [0.5, 1, 2, 4] meters.
Hierarchical Mean Average Precision (m.P For safety critical applications, although correctly localizing and classifying object is ideal, detecting but misclassifying some object is more desirable than a missed detection (e.g., detect but misclassify a chi I as adult versus not detecting this child). Therefore, we introduce hierarchical AP (APH) which considers such semantic relationships

---

across classes to award partial credit. To encode these relationships between classes, we leverage the semantic hierarchy (Fig. 3) defined by nuScenes. We derive partial credit as a function of semantic similarity using the least common ancestor (LCC) distance metric. Hierarchical metrics have been proposed for image classification [44], but have not been explored for object detection. Extending this metric for object detection is challenging because we must consider how to jointly evaluate semantic and spatial overlap. For clarity, we will describe the procedure in context of computing AP и for some arbitrary class C.
LCA=0:
Consider the predictions and ground-trutt -runt boxes for C. Label the set of predictions that overlap with ground-truth boxes for C as true positives. Other predictions are false positives. This is identical to the standard AP metric.
LCA=1:
Consider the predictions for C, and ground-truth boxes for C and all sibling classes of C (that have LCA distance C of I). Label the set of predictions that overlap a ground-truth box of C' as a true positive. Label the set of predictions that overlap sibling classes as gnnreed [43]. All other predictions for C' are false positives.
LCA=2: Consider the predictions for and ground-truth boxes for C' and all sibling classes of C (that have LCA distance to C less than 2. For nuScenes, this includes all classes.) Label the set of predictions that overlap ground-trun truth boxes for C' as true positives. Label the set of predictions that overlap other classes ignored. All other predictions for C' are false positives.
Experiments
We conduct experiments to better understand the LT3D problem, and gain insights by validating our techniques described in Section 3. Specifically, we aim to answer the following questions:'
1. Are rare classes more difficult detect than common classes?
2. Are objects from rare classes sufficiently localized but mis-classified?
3. Does training with the semantic hierarchy improve detection performance for LT3D?
4. Does multimodal fusion help detect zare classes?
We start this section by introducing the model architecture, implementation and dataset.
Model Architecture. For LiDAR-based 3D detectors, we PointPillars [8] and CenterPoint [13], which are widely benchmarked in the literature. For fusion-based 3D detectors, we use TransFu- sion [17], a recently released state-of-the-ar method. TransFusion proposes an end-to-end DETR-like approach [45] for multimodal 3D detection.
Implementation. We the PyTorch toolbox [46] train all models for 20 epochs with the Adam optimizer [47] and a one-cycle learning rate scheduler [48]. In training, we adopt data augmentation techniques introduced by [13]. When using the introduced data augmentation schedule (cf. Section 3), we train models for 15 epochs with data augmentation enabled, and 5 epochs without, We further describe our implementation in the supplement.
Datasets. We use nuScenes [2] and Argoverse 2.0 (AV2) [49] to explore LT3D. Both have fine-grained classes (18 and 26 classes in nuScenes and AV2 respectively) that follow long-tailed distributions. To quantify the long-tail, we use the imbalance factor (IF) defined as the ratio between the numbers of annotations of the max-class and min-class [32]; nuScenes and AV2 have IF=1670 and 2500 respectively significantly more imbalanced than existing long-tail image recognition benchmarks, e.g.. iNaturalist (IF=500) [50] and ImageNet-LT (1F=1000) [51]. NuScenes arranges classes in semantic hierarchy (Fig. 3): AV2 does not provide a semantic hierarchy but we construct one based on the nuScenes' hierarchy. Following prior work, we use their official train-sets for training and evaluate on the their official val-sets. We focus on nuScenes in the main paper and AV2 in the supplement. Our primary conclusions hold for both datasets.
Retraining state-of-the-art 3D detectors for LT3D. We retrain several 3D detectors, namely FCOS3D [41], PointPillars [8] and CenterPoint [13]. FCOS3D operates monocular images. The other three detectors take an aggregated stack of ten LiDAR-sweeps as input. All four models predict 3D bounding boxes for 18 classes defined by the nuScenes LT3D protocol. As shown in
Answwrs: yes, yes, yes, yes.
5

---

Objoca (nancl
Objects
Vettalee
Pedretrikns
Morable
Pederriin
Mavale
Olr
Bus "ruuk
Adulb
Rermer Traftle Ocse
Ootatrraciin Valtitil
Polzce octece Construction Worisee
Matooegrolo
Bogyela
Breoliee Parsonal Mobulity
Dehri Pushable Pullable
Sindfer
Traile
Rmergency Veitiole
Figure 3: nuScenes defines a semantic hierarchy (on the left) for all annotated classes (Fig. 1). We highlight common classes in white and rare classes in gold. The standard nuScenes benchmark makes two choices for dealing with classes: (1) ignore them (c.g., stroller and pushable-pullatt them into coarse-gras goarse-grained classes (c.g., adult, child, grouped as pedestris rin),. Since the pedestrian class is dominated by adult (Fig. 1), the standard benchmarking protocol masks the challenge of detecting rare classes like chi 1d and police-ol oo oofffff fifffii cer. We leverage this hierarchy during training (on the right) by predicting class labels multiple levels of the hierarchy. Specifically, train detectors predict three lahels for each object: its fine-grained label (e-g., child, its coarse class (e-g., pedest rin), and the root-level class ob ject. This means that final vocabulary of classes is no longer mutually exclusive, complicating the application of multi-class softmax losses. To address this, use sigmoid focal loss that learns separate spatial heatmaps for each class.
(2) group
wenttruuctioowworoee. pelice-off feer are
Table I: Benchmarking detectors for LT3D (measured by mAP). We present several salient conclusions. First, training with the semantic hierarchy improves all methods for LT3D, e.g., improving by 1% AP averaged over All classes. Second, multimodal filtering yields between 4~~1 AP improvement on Medi and Few classes! This surprising given that FCOS3D is less powerful 3D detector its own. Importantly, we do not modify FCOS3D for LT3D. Interestingly, it also improves multimodal detectors (1.6% AP improvement for TransFusion on All classes), demonstrating the importance of using RGB to improve LT3D with better recognition. Third, perhaps surprisingly, post-hoc multimodal filtering of LiDAR-only detector CenterPoint with RGB-only detector FCOS3D performs the best, surpassing the multi-modal TransFusion model. Lastly, data augmentation schedules detection in the long-tail.
necessarily improve LT3D performance, demonstrating challenge of 3D
Method
Multimodal
Many
Medium
Few
A11
FCOS3D (RGB-only) [41]
39,0
23.3
2.9
20.9
PointPillars (LiDAR-only) [8]
64.2
28.4
3.4
30.0
+ Hierarchy
66.4
30.4
2.9
31.2
w/ Data Aug,
54.4
24.2
1.8
25.1
w/ Multimodal Filtering
66.2
41.0
4.4
35.8
CenterPoint (LiDAR-only) [13]
76.4
43.1
3.5
39.2
+ Hicrarchy
77.1
45.1
4.3
40.4
Data Aug.
73.8
44.5
7.4
40.3
w/ Multimodal Filtering
77.1
49.0
9.4
43.6
TransFusion-L (LiDAR [17]
68.5
42.8
8.4
38.5
w/ Multimodal Filtering
73.2
42.5
8.3
39.6
TransFusion (LiDAR RGB)
73,9
41.2
9.8
39.8
w/ Data Aug.
73.4
40.9
8.2
39.0
Multimodal Filtering
73.9
42.5
9.1
40.1
Table 1, LiDAR- detectors that perform well common classes tend to also perform well on rare classes.
Training with Semantic Hierarchy. Next, we modify our LiDAR- detectors to jointly predict class labels different levels of the semantic hierarchy. For example, we modify the detector to additionally classify st roller as pedest r ian and ob ject. The semantic hierarchy naturally groups classes based on shared attributes and may have complementary features. Moreover, training with the semantic hierarchy allows rare classes within each group learn better features by sharing with common classes. This approach generally effective, as shown in Table improving accuracy for classes with Many examples by 2%, Medium examples by 2% and Few examples by 1% AP.

---

Table 2: Diagnosis using mAP# metric on selected classes. We analyze the best-performing LiDAR- only model CenterPoint and multimodal model TransFusion, with / without our hierarchical loss (hier.) and mutimodal filtering technique (fittering) Comparing the rows of LCA=0 for with and without our techniques (for CenterPoint and TransFusion respectively). we our techniques bring significantly improvements classes with medi um and fow examples such as construct www.stttotto-oooceeeoo (CV), bicyele, motoreyele (MC), constr tentuuucttt on cnwworker (CW), stroller: and eessaable-lllallle (PP). Moreover, performance increases significantly from LCA=0 to ICA=1 compared against ICC==1 to LCA=2, (Table 2), confirming that objects from rare classes are often detected hut misclassified as some sibling classes.
Method
m.Pu
Car
Adult
Truck
CV
Bieycle
MC
Child
CW
Stroller
PP
LCA=0
82.4
81.2
49.4
19.7
33.6
48.9
0.1
14.2
0.1
21.7
CenterPoint (OTS)
LCA=1
83,9
82.0
58.7
20.5
35.2
50.5
18.3
22.0
LCA=2
84.0
82.4
58,8
20.7
36.4
51.0
0.1
19.5
0.1
22.6
LCA=0
88.1
86.3
62.7
24.5
48.5
62.8
22.2
4.3
32.7
CenterPoint (Group-Free)
LCA=1
89.0
87.1
71.6
26.7
50.2
64.7
0.1
29.4
4.5
32.9
LCA=2
89.1
87.5
71.7
26.8
51.1
65.2
0.1
30.5
4.8
33.4
LCA=0
88.6
86.9
63.4
25.7
50.2
63.2
0.1
25.3
8.7
36.8
CenterPoint (Group-Free
TCC=1
89.5
87.6
72.4
27.5
52.2
65.2
0.1
32.4
9.4
37.0
w/ Hienarcky
LCA=2
89.6
88.0
72.5
27.7
53.2
65.7
0.1
34.0
9.8
37.6
LCA=0
88.5
86.6
63.4
29.0
58.5
68.2
5.3
35.8
31.6
39.3
CenterPoint (Group-Free
LCA=1
89.4
87.4
72.4
31.3
61.2
69.7
15.2
52.0
37.7
39.4
w/ Hier. & MM. Filtering
LCA=2
89.5
87.7
72.5
31.5
62.3
69.9
16.9
56.3
38.8
39.8
LCA=0
84.4
84.5
58.5
15.1
44.9
57.2
1.0
15.1
3.2
19.6
TransFusion-L
LCA=1
85.5
85.7
67.4
21.8
46.7
59.1
1.6
21.8
3.7
19.8
(LiDAR-only)
LCA=2
85.5
86.1
67.5
22.6
47.7
59.9
1.7
22.6
4.2
20.4
LCA=0
84.4
84.2
58.4
24.5
46.7
60.8
3.1
21.6
13.3
25.3
TransFusion
LCA=1
86.0
85.4
67.3
26.3
50.1
63.5
14.4
34.7
20.6
25.6
(LiDAR RGB)
LCA=2
86.0
85.9
67.4
26.8
52.2
65.1
15.2
36.1
22.8
26.4
LCA=0
84.4
84.2
58.4
25.3
52.3
62.8
4.0
27.5
14.7
27.3
TransFusion
LCA=1
86.0
85.4
67.3
26.6
55.7
64.0
25.1
46.7
24.3
27.4
w/ Multimodal Filtering
LCA=2
86.0
85.9
67.4
27.0
56.9
64.3
25.8
48.6
28.3
27.9
Data Augmentation Schedule. Prior works [25, 17] suggest disabling copy-paste augmentation for the last few epochs of training reduce the number of false positive detections. We validate this claim for various detector architectures and find that although it seems to help rare classes by 3% AP, but hurts common classes by 4% AP (c.f. CenterPoint).
Multimodal Fusion via Filtering. Detecting rare classes from LiDAR-only is challenging since its difficult to recognize objects from sparse LiDAR points alone. As result, have many high-scoring FPs (Fig. 2), resulting in low AP. Using RGB detections to filter the LiDAR detections results in significant performance improvement rare classes, improving by 4~ AP on classes with Medi um and Few examples for all models (Table 1)!
often
End-to-End Multimodal Methods. Since multimodal cues significantly improve LT3D, we are motivated explore end-to-end approaches. Specifically, we retrain TransFusion [17] on all 18 classes. We retrain TransFusion, downsampling the RGB images by a factor of 2 to fit the model in GPU memory. Surprisingly, lidar-only and multi-modal variants of TransFusion perform worse than CenterPoint for L.T3D. Further hyperparameter tuning and full-resolution training may help. Lastly, our multimodal filtering strategy still improves this end-to-end fusion methods slightly, e.g., increases mAP on All classes by 0.3% AP for TransFusion (cf. Table 1). We also find that applying multimodal filtering on the LiDAR-onDe branch TransFusion yields similar performance to the multimodal model, suggested that TransFusion is simply learning multimodal filtering function.
Analysis of Misclassifications. For 3D detection, localization and classification are two important measures of 3D detection performance. In practice, we cannot achieve perfect performance for either. In safety-critical applications, detecting but misclassifying objects (as semantically related category) is more desirable than missed detection (e.g., detect but misclassify chi mill as adult versus not detecting this chi 1d). Therefore, we introduce hierarchical AP (AP,r), which considers such semantic relationships across classes to award partial credit. Applying this hierarchical AP reveals
7

---

Vehicle clc
Pedes st r an
Movable
221
Gdt
3
8
221
602
035
002
8
OC2
9:34
0.9
F21
po ade polall
Bi Baroee
in vetr
407
fel ldla
contriicce Mastee
persenal sealitly
(g) imolle
Figure 4: Breakdown analysis of misclassifications within superclasses. We analyze our best-performing model (CenterPoint w/ hierarchical training and multimodal filtering). Fine-grained classes are most often confused by the dominant class (in blue) in each superclass: (left) Vehicle is dominated by car, (mid) Pedest rian rian dominated by adult, t, and (right) Movable dominated by barrier. We find that class confusions are reasonable. Car is often mistaken for truck. Similarly, truck, construct, -on-veh veli and eme weeeer-eeeeeecccooo misclassified car, presumably because they are sometimes spatially close (within the 2m match threshold) cars. Adults have similar appearance police-ol col-cc--fffiooeeee afffcces cer and const on--woree: and they are often co-localized with child and stroller; all of these might cause significant class confusion.
are most often mistaken for car. Bicycle and mot orcycle are sometimes
that classes are most often misclassified as their LCA=1 siblings within coarse-grained superclasses. We use confusion matrices to further analyze the misclassifications within superclasses, as shown in Fig, 4. Below, we explain how to compute confusion matrix for the detection task.
For each superclass, we make confusion matrix, in which the entry (i. indicates the misclassifica- tion rate of class-i objects as class-j. Specifically, given a fine-grained class i, we find its predictions that match ground-truth boxes within 2m center-distance of class-i and all its sibling classes (LCA=I, within the corresponding superclass); we ignore all unmatched detections. This allows us to count the missclassifiaatoons of class-i objects into class-j, with which simple normalization produces misclassification rates.
Limitations. LT3D emphasizes object detection for rare classes which can be safety-critical for downstream AV tasks such as motion planning and collision avoidance. However, our work does not study how solving LT3D directly affects these tasks. Future work should address this limitation. Another limitation, shared by contemporary benchmarks, is that our setup does not consider the correlation between individual classes. For example, the rare-class st mrollee often pushed by an adult. lt. One may argue that detecting adult is sufficient for safe navigation. However, edge cases can occur in the real world where a stroller can be unattended.
Conclusion
We explore the problem of long-tailed 3D detection (LT3D), detecting objects not only from common classes but also from many zare classes. This problem is motivated by the operational safety of autonomous vehicles (AVs), but has broad robotic applications, e.g., elder-assistiv robots [52] that fetch diverse items [53] should address LT3D. To study LT3D, we establish rigorous evaluation protocols that allow for partial credit to better diagnose 3D detectors. We propose several algorithmic innovations to improve LT3D, including group-free detector head, hierarchical losses that promote feature sharing across long-tailed classes, and a simple multimodal fusion method that uses RGB- based detections to filter LiDAR-based detections, achieving significant improvement for LT3D.
Acknowledgement
This work was supported by the CMU Argo AI Center for Autonomous Vehicle Research.
8

---

References
[1] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
[2] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[3] M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2019.
[4] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam, H. Zhao, A. Timofeev, S. Ettinger, M. Krivokon, A. Gao, A. Joshi, Y. Zhang, J. Shlens, Z. Chen, and D. Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[5] A. Taeihagh and H. S. M. Lim. Governing autonomous vehicles: emerging responses for safety, liability, privacy, cybersecurity, and industry risks. Transport Reviews, 39(1):103-128, 2019.
[6] K. Wong, S. Wang, M. Ren, M. Liang, and R. Urtasun. Identifying unknown instances for autonomous driving. In CoRI.,
[7] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu. Large-scale long-tailed recognition in an open world. In JEEE Conference on Computer Vision and Pattern Recognition ( (CVPR), 2019.
[8] A. H. Lang, S. Vora, H. Caesar, L. Zhou, Yang, and O. Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In IEEE Conference on Computer Vision and Patlern Recognition (CVPR), 2019.
[9] B. Zhu, Z. Jiang, X. Zhou, Z. Li, and G. Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint 1rX0v:1908.00492, 2019.
[10] P. Hu, J. Ziglar, D. Held, and D. Ramanan. What you see is what you get: Exploiting visibility for 3d object detection. CoRR, abs/1912.04999 2019.
[11] Y. Yan, Y. Mao, and B. Li. Second: Sparsely embedded convolutional detection. Sensors, 18 (10):3337, 2018.
[12] G. Wang, Y. Wang, H. Zhang, R. Gu, and J.-N. Hwang. Exploit the connectivity: Multi- object tracking with trackletnet. In Proceedings of the 27th ACM International Conference on Multimedia, pages 482-490) 2019.
[13] T. Yin, X. Zhou, and P. Krähenbühl. Center-based 3d object detection and tracking. arXiv preprint arXiv:2006. / 1275, 2020.
[14] X. Chen, H. Ma, Wan, Li, and T. Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017.
[15] Ku, M. Mozifian, J. Lee, A. Harakeh, and Waslander. Joint 3d proposal generation and object detection from view aggregation. IROS, 2018
[16] J. H. Yoo, Y. Kim, J. S. Kim, and J. W. Choi. 3d-evf: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection. In European Conference on Computer Vision (ECCV), 2020.
[17] X. Bai, Z. Hu, X. Zhu, Q. Huang, Y. Chen, H. Fu, and C. Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. CoRR, abs/2203. 2022.
[18] Y.-T. Chen, Shi, Z. Ye, C. Mertz, D. Ramanan, and S. Kong. Multimodal object detection via probabilistic ensembling. In European Conference on Computer Vision (ECCV), 2022.

---

[19] S. Gupta, J. Kanjani, M. Li, F. Ferroni, J. Hays, D. Ramanan, and S. Kong. Far3det: Towards far-field 3d detection. NeurIPS, 2022.
[20] V. A. Sindagi, Y. Zhou, and O. Tuzel. Mvx-net: Multimodal voxelnet for 3d object detection. In International Conference on Robotics and Automation (ICRA). 2019.
[21] Y. You, Y. Wang, W.-L. Chao, D. Garg, G. Pleiss, B. Hariharan, M. Campbell, and K. Q. Weinberger. Pseudo-lidart+:: Accurate depth for 3d object detection in autonomous driving. arXiv 190iv 06310, 2020.
[22] S. Vora, A. H. Lang, B. Helou, and O. Beijbom. Pointpainting: Sequential fusion for 3d object detection. In IEEE/CVE conference on computer vision and pattern recognition, 2020.
[23] T. Yin, X. Zhou, and P. Krähenbühh.. Multimodal virtual point 3d detection. NeurIPS, 2021.
[24] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas. Frustum pointnets for 3d object detection from rgb-d data. In IEEE conference on computer vision and pattern recognition, 2018.
[25] D. Xu, D. Anguelov, and A. Jain. Pointfusion: Deep sensor fusion for 3d bounding box estimation. In IEEE conference computer vision and pattern recognition, 2018.
[26] S. Pang, D. Morris, and H. Radha. Cloes: Camera-lida -idar object candidates fusion for 3d object detection. In /EEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
[27] W. Reed. The pareto, zipf and other power laws. Economics letters, 74(1):15-19, 2001.
[28] Y. Zhang, B. Kang. B. Hooi, S. Yan, and J. Feng. Deep long-tailed learning: A survey. 1xXv:2210004496 2021.
[29] S. Alshammari, Y.-X. Wang, D. Ramanan, and S. Kong. Long-tailed recognition via weight balancing. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
[30] Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie. Class-balanced loss based on effective number of samples. In CVPR, 2019.
[31] S. H. Khan, M. Hayat, M. Bennamoun, F. A. Sohel, and R. Toggneii Cost-sensitive learning of deep feature representations from imbalanced data. IEEE transactions on neural networks and learning systems, 29())33573-3587 2017.
[32] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma. Learning imbalanced datasets with label-distriuution-aawe awware margin loss. In NeurlPS, 2019.
[33] S. Khan, M. Hayat, S. W. Zamir, Shen, and L. Shao. Striking the right balance with uncertainty. In CVPR. 2019.
[34] C. Huang, Y. Li, C. C. Loy, and X. Tang. Deep imbalanced learning for face recognition and attribute prediction. PAMI, 42 ((11):2781-277 2019.
[35] S. Zhang, Z. Li, S. Yan, X. He, and J. Sun. Distribution alignment: A unified framework for long-tail visual recognition. CVPR, 2021.
[36] C. Drummond, R. C. Holte, et al. C4. 5, class imbalance, and cost sensitivity: why under- sampling beats over-sampling. In Workshop learning from imbalanced datasets II, 2003.
[37] N. V. Chawla, W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321-357, 2002.
[38] H. Han, W.- Wang, and B.-H. Mao. Borderline-smote: new over-sampling method in imbalanced data sets learning. In International Conference on Intelligent Computing, pages 878-887. Springer, 2005.
[39] K. Tang. J. Huang, and H. Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In NeurIPS, 2020.
10

---

[40] A. Gupta, P. Dollar, and R. Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR. 2019.
[41] T. Wang. X. Zhu, J. Pang, and D. Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In /EEE/CVF International Conference on Computer Vision, pages 913-922, 2021.
[42] M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision (IJCV), 2015.
[43] T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: common objects in context. In European Conference on Computer Vision (ECCV), 2014.
[44] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):11-222.2 2015.
[45] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV), 2020.
[46] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, I., Antiga, A. Desmaison, A. Köpf, E. Yang. 7. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, I.. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
[47] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
[48] L. N. Smith. Cyclical learning rates for training neural networks. In IEEE Winter Conference on Applications of Computer Vision, WACV, 2017.
[49] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hart- nett, J. K. Pontes, D. Ramanan, P. Carr, and J. Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[50] G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
[51] Z.. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu. Large-scale long-tailed recognition an open world. In CVPR, pages 2537-2546, 2019.
[52] N. Savage. Robots rise to meet the challenge of caring for old people. Nature, 2022.
[53] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang. M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, Bansal, D. Batra, V. Cartillier, S. Crane. T. Do.
M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q. Fu, C. Fuegen, A. Gebreselasie, Huang, W. Jia, W. Khoo, Kolar, S. Kottur, A. Kumar, F. Landini, C. Li, Y. Li, Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, R. Puentes, M. Ramazanova, L. Sari, K. Somasundaram, A. Southerland, Sugano, R. Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Y. Zhu, P. Arbelaez, D. Crandall, D. Damen, G. M. Farinella, B. Ghanem, V. K. Ithapu, C. V. Jawahar, H. Joo, K. Kitani, H. Li, R. Newcombe, A. Oliva, H. S. Park, J. M. Rehg. Y. Sato, J. Shi, M. Z. Shou, A. Torralba, L. Torresani, M. Yan, and J. Malik. Ego4d: Around the world in 3, 000 hours of egocentric video. Computer Vision and Pattern Recognition 2022.
C. Gonzalez, Hillis, X. Huang.
[54] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In ICCV, 2017.
11

---

[55] S. Ren, K. He. R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 2015.
[56] C. J. Wu, M. Tygent and Y. LeCun. A hierarchical loss and its problems when classifying non-hierarchically PLoS ONE, 14, 2019.
[57] Y. Li, T. Wang, B. Kang, S. Tang, C. Wang. J. Li, and J. Feng. Overcoming classifier imbalance for long-tail object detection with balanced group softmax. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[58] A. E. Welchman, L. Tuck, and J. M. Harris. Human observers are biased in judging the angular approach of projectile. Vision research, 44(17):007-2042, 2004.
[59] T. Wang, X. Zhu, J. Pang, and D. Lin. FCOS3D: fully convolutional one-stage monocular 3d object detection. In JEEE/CVE International Conference on Computer Vision Workshops, 2021.
12


Reviews:
[{"review_id": "ZhZAmx3HG33", "reviewer": "Reviewer_LdEV", "summary": "This paper studies the problem of 3D detection and classification of rare objects in urban driving. It proposes to use hierarchical classification to detect rare objects even if they may not be classified correctly. The authors develop a method to train hierarchical 3D object detectors from either RGB or LIDAR and an evaluation protocol to measure detection performance under hierarchies. Evaluation is performed on the real-world nuScenes dataset, but no robot demonstration or validation in deployment is offered.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**strengths**\n\n- Hierarchical classification and long-tail detection are underexplored and relevant problems in robotic perception, beyond the studied application of autonomous driving.\n- The proposed methodology is evaluated across a range of different architectures and methods, showing a general effectiveness.\n- The paper is very well written and the detailed evaluation protocol is of particular value to spark future work in this area.\n- The multimodal filtering trick is simple yet effective, even when applied to methods that are already trained to fuse modalities.\n\n**weaknesses**\n\n- No improvement over existing methods for hierarchical classification. It seems that actually most of the tested existing methods have better long-tail performance than the proposed method.\n- There are no robotic experiments. This is a computer-vision paper, but with high relevance to robotic perception. \n\n**points resolved through the revision**\n- The authors do not describe how their models are trained. They describe in general that they do not use a 'softmax loss' (I assume they mean cross-entropy), but don't describe what they use instead. They also do not describe how the RGB methods are altered for hierarchical classification.\n- There is no comparison to related work in hierarchical classification. It is unclear whether the hierarchical training is novel or useful compared to existing methods for training hierarchical classification (that exist, according to the related work section).\n- Given the lack of implementation detail in the manuscript and no code supplement, the method and the results are not reproducible. (The authors claim to open-source their code later on, but for CoRL authors should use the offered option of the code supplement or describe their method better in the text.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "**summary before revision:**\nI think this paper pushes into the right direction, emphasising a number of problems with current detection methods (not only in 3D) and coming up with sensible ideas. It is valuable to discuss these ideas and problems at CoRL. The biggest weakness of this paper is that it is unclear to me how much of the proposed method is original and how much adapted from problems like image classification. Further, the results are not reproducible due to lack of details. Both issues however can be rather easily addressed through discussion with the authors and revision.\n\n**summary after revision:**\nI think this paper pushes into the right direction, emphasising a number of problems with current detection methods (not only in 3D) and coming up with sensible ideas. It is valuable to discuss these ideas and problems at CoRL. The comparison with existing hierarchical classification works however shows no improvement of this particular method. Existing methods even seem to perform better in the long-tail classes. There are still many valuable ideas and approaches in this paper, but this comparison makes the contribution less strong than I initially thought.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Most importantly, I recommend the authors to address the points listed under 'weaknesses'. \n\nAdditionally, here are more detailed notes of issues I found in the submission:\n\n- The abstract claims that you would propose \"hierarchical losses\". The losses that you use are not described, but from context it seems to me that these losses actually do not enforce any hierarchy, but just enable a detection to have multiple class-assignments (that do not necessarily have to have a hierarchy).\n- You claim to boost performance in the problem of 3D object detection (line 55). I think this claim goes too far, as it implies that your training is better than the established approaches on nuscenes. For this, a comparison to the standard training (with the coarser classes) would be necessary. It would actually be interesting to see if the added information of more fine-grained classes boosts performance compared to training with only coarse classes. For the current state of the paper, a formulation like \"compared to naive training on the long-trailed class distribution\" is necessary in line 55.\n- line 73: typo (RGB)\n- line 122: Here, please explain in detail which loss you use and why. How does it relate to existing works on hierarchical classification? \n- line 122: Personally, I find 'cross-entropy loss' more clear than 'softmax loss'.\n- line 127: This is pure speculation. Please remove or provide an argument.\n- line 168: I recommend that you explain more clearly how your proposed metric relates to [43]. Is it a similar methodology of LCA levels, just applied to a different base metric? something entirely different?\n- 19 classes does not seem particularly long-tailed compared to e.g. class distributions in Mapillary Vistas. I understand the argument that the rare classes are tail-classes because they are orders of magnitude less frequent than the majority classes. However, I think it would help to quantify and relate this: How many orders of magnitude are between tail classes and majority classes in other datasets? Are results from the nuScene dataset with only 19 classes relevant to long-tailed classification in general?\n- lines 216ff: Please discuss your training approach in relation to other works. Why did you follow this approach? How does it relate to existing hierarchical classification approaches? What is novel about your approach?\n- line 216: Do you also modify the RGB-based detectors?\n- Table 1: Is '+Camera' for TransFusion a typo? If not, what does it mean? Do you not train TransFusion with hierarchies?\n- lines 240-241: Why did you downsample instead of using crops during training? Does a single image for inference not fit into memory? Also, please indicate in all tables that TransFusion has less information available (downsampling makes detection of far-away objects harder) compared to your multimodal filtering approach.\n- Table 2: Can you please add here a comparison with a detector just trained on the coarse training protocol from nuScenes (basically trained directly only on the LCA=1 level)? The same could be done with LCA=2. Both would give a good indication of what established training methods would achieve for this evaluation level, and give context what a potential 'upper bound' is.\n"}}, {"review_id": "D-h5ZV-L0HX", "reviewer": "Reviewer_bNMv", "summary": "This paper studies the Long-Tailed 3D Detection (LT3D) problem in the application domain of autonomous vehicles (AVs). It argues that accurately detecting long-tailed classes in AV datasets is crucial to their safe and effective deployment in real-world applications. Most current benchmarks, however, either: (a) do not measure the performance of state-of-the-art detectors on these long-tailed classes, or (b) group them with other, more commonly occurring classes, inhibiting detector performance on these classes. The paper addresses these concerns by proposing architectural improvements, such as training with semantic hierarchies and multi-modal filtering. It also presents a hierarchical evaluation metric for measuring the performance of a detector on all classes, including long-tailed classes. The experimental evaluation evaluates different detectors on the nuScenes dataset according to the proposed metric.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "On the positive side, the paper is well-written and the problem of LT3D is clearly defined. The experimental section is thorough in terms of the detection methods being evaluated. It evaluates both multi-modal and non-multi-modal detectors, which is appreciated. It also presents an ablation study of the different detector components proposed in Section 3. Table 1 summarizes the state-of-the-art in this problem domain and highlight the key insights from different detection methods. It presents clearly the technical improvements proposed to state-of-the-art detectors and the evaluation metrics designed to measure LT3D performance.  The survey of related work also appears to be representative of some of the related work.\n\nIn terms of limitations, the main concern is that there has been a lot of work in long-tailed recognition, especially for the 2D case, with similar ideas to those proposed in the current paper on how to address this challenge (e.g., hierarchical reasoning). Some of these efforts are mentioned in Section 2. The paper does not do a good job at this point, however, of highlighting how the current work differentiates from them beyond trying the same ideas in the 3D domain.  Stronger justification beyond “...rare classes are not only infrequent, but are also difficult to distinguish using LiDAR alone” would be helpful to emphasize the importance of the challenge in this domain. Because the proposed solution appears to be rather standard given the literature and it is not clear how the nature of the LiDAR data affects the approach. \n\nWhile there is some improvement in detection using hierarchies and multimodal filtering (Table 2), it does not reach the point of sufficiently supporting the claim that these solutions are addressing the challenge. For instance, the mAP_H values for the \"Child\" and \"Pushable-Pullable\" classes are still rather low.\n\nFurthermore, the experimental evaluation only considers a single dataset, nuScenes. The argument made in Section 5.1 is that nuScenes explicitly defines the semantic hierarchies required for the evaluation protocol proposed in the paper. Nevertheless, it should be possible to use external semantic hierarchies (e.g., WordNet) for other dataset labels.\n\nIn Section 3, Line 110, it is stated that \"...heads of rare classes overfit..\". Then, in Section 3, Line 120, \"...use a per-class logit-loss rather than multi-class softmax loss...\". Why does the latter not suffer from the same overfitting problem for rare classes.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well written and the technical direction for approaching the problem of LT3D is sound, while there are accompanying experiments that use state-of-the-art detectors. The novelty, however, is not clear given the amount of work in long-tailed detection and the manuscript does not sufficiently differentiate the challenge from the work in 2D detection. The arguments would also be better supported if the proposed changes were evaluated on more than a single dataset, where it may be possible to argue stronger improvements.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Better support the novelty and significance of the work in context of prior and ongoing work in the area of computer vision algorithms focusing on long-tailed detection.\n\nExperiments on additional datasets. The semantic hierarchy can be manually handcrafted. It would be acceptable to evaluate only the CenterPoint and Transfusion detectors.\n"}}, {"review_id": "4Mji9tDPAJr", "reviewer": "Reviewer_4cQJ", "summary": "This paper studies a very important problem in autonomous driving: long-tail 3D object detection. The authors propose several useful training methods and did an in-depth evaluation, which gives the readers some valuable insights.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n1. The in-depth analysis in both the method and experiment section provides the readers some very useful insights on training 3D object detectors in long-tail classes, such as using group-free detection heads and training with semantics hierarchies without softmax classification. \n\n2. The question “Are objects from rare classes sufficiently localized but mis-classified” is very valuable. The authors claimed and showed that the LiDAR detector can reach a high recall in terms of 3D localization, but poor at classification. Then the authors use RGB modality to filter out the false positives, and improve the classification performance. \n\n### Weaknesses:\n\n1. The paper does not propose a significantly novel framework. But the analysis and insights are interesting.\n\nIn summary, the paper provides a valuable analysis on training 3D object detectors for long-tail classes. I recommend for publication because this paper could potentially inspire new methods.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Recommend for publication", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "What if the lidar detector gives many false positives along a single line of sight on the RGB image? How can the RGB-based detector classifiy which one is the most correct?", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "0-1yXiRkfUk", "reviewer": "Reviewer_b6Vx", "summary": "Existing methods for 3D classification of objects for autonomous vehicle benchmarks neglect rare classes, classes in the tail, and also ignore the hierarchies that exists among the labels, such as \"child\" and \"construction-worker\" are sub-classes of \"pedestrian\". The paper argues that multi-modal approaches, fusing RGB images with LiDAR deal better with the long-tailed 3D detection. Besides, the paper introduces a detection metric that favors mistakes that are less riskier, e.g., mistaking a child for an adult. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n* The paper introduces some techniques that seems to be useful to improve classification accuracy both on common and rare classes. \n* The techniques introduced in the paper are simple to implement without adding too much computational burden. \n\nWeakness: \n* This is a very limited study which is unlikely to have robotic applications other than autonomous vehicles. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper in general seems to provide a simple method and convincing results. However, the relevance to the robotic field is questionable. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Irrelevant to robotics", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "The paper looks fine. ", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)