Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Existing methods for 3D classification of objects for autonomous vehicle benchmarks neglect rare classes, classes in the tail, and also ignore the hierarchies that exists among the labels, such as "child" and "construction-worker" are sub-classes of "pedestrian". The paper argues that multi-modal approaches, fusing RGB images with LiDAR deal better with the long-tailed 3D detection. Besides, the paper introduces a detection metric that favors mistakes that are less riskier, e.g., mistaking a child for an adult. 
strengths_and_weaknesses: Strength: 
* The paper introduces some techniques that seems to be useful to improve classification accuracy both on common and rare classes. 
* The techniques introduced in the paper are simple to implement without adding too much computational burden. 

Weakness: 
* This is a very limited study which is unlikely to have robotic applications other than autonomous vehicles. 
robotics_focus: Irrelevant to robotics
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper in general seems to provide a simple method and convincing results. However, the relevance to the robotic field is questionable. 
issues: The paper looks fine.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper studies the problem of 3D detection and classification of rare objects in urban driving. It proposes to use hierarchical classification to detect rare objects even if they may not be classified correctly. The authors develop a method to train hierarchical 3D object detectors from either RGB or LIDAR and an evaluation protocol to measure detection performance under hierarchies. Evaluation is performed on the real-world nuScenes dataset, but no robot demonstration or validation in deployment is offered.
strengths_and_weaknesses: **strengths**

- Hierarchical classification and long-tail detection are underexplored and relevant problems in robotic perception, beyond the studied application of autonomous driving.
- The proposed methodology is evaluated across a range of different architectures and methods, showing a general effectiveness.
- The paper is very well written and the detailed evaluation protocol is of particular value to spark future work in this area.
- The multimodal filtering trick is simple yet effective, even when applied to methods that are already trained to fuse modalities.

**weaknesses**

- No improvement over existing methods for hierarchical classification. It seems that actually most of the tested existing methods have better long-tail performance than the proposed method.
- There are no robotic experiments. This is a computer-vision paper, but with high relevance to robotic perception. 

**points resolved through the revision**
- The authors do not describe how their models are trained. They describe in general that they do not use a 'softmax loss' (I assume they mean cross-entropy), but don't describe what they use instead. They also do not describe how the RGB methods are altered for hierarchical classification.
- There is no comparison to related work in hierarchical classification. It is unclear whether the hierarchical training is novel or useful compared to existing methods for training hierarchical classification (that exist, according to the related work section).
- Given the lack of implementation detail in the manuscript and no code supplement, the method and the results are not reproducible. (The authors claim to open-source their code later on, but for CoRL authors should use the offered option of the code supplement or describe their method better in the text.)
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: **summary before revision:**
I think this paper pushes into the right direction, emphasising a number of problems with current detection methods (not only in 3D) and coming up with sensible ideas. It is valuable to discuss these ideas and problems at CoRL. The biggest weakness of this paper is that it is unclear to me how much of the proposed method is original and how much adapted from problems like image classification. Further, the results are not reproducible due to lack of details. Both issues however can be rather easily addressed through discussion with the authors and revision.

**summary after revision:**
I think this paper pushes into the right direction, emphasising a number of problems with current detection methods (not only in 3D) and coming up with sensible ideas. It is valuable to discuss these ideas and problems at CoRL. The comparison with existing hierarchical classification works however shows no improvement of this particular method. Existing methods even seem to perform better in the long-tail classes. There are still many valuable ideas and approaches in this paper, but this comparison makes the contribution less strong than I initially thought.
issues: Most importantly, I recommend the authors to address the points listed under 'weaknesses'. 

Additionally, here are more detailed notes of issues I found in the submission:

- The abstract claims that you would propose "hierarchical losses". The losses that you use are not described, but from context it seems to me that these losses actually do not enforce any hierarchy, but just enable a detection to have multiple class-assignments (that do not necessarily have to have a hierarchy).
- You claim to boost performance in the problem of 3D object detection (line 55). I think this claim goes too far, as it implies that your training is better than the established approaches on nuscenes. For this, a comparison to the standard training (with the coarser classes) would be necessary. It would actually be interesting to see if the added information of more fine-grained classes boosts performance compared to training with only coarse classes. For the current state of the paper, a formulation like "compared to naive training on the long-trailed class distribution" is necessary in line 55.
- line 73: typo (RGB)
- line 122: Here, please explain in detail which loss you use and why. How does it relate to existing works on hierarchical classification? 
- line 122: Personally, I find 'cross-entropy loss' more clear than 'softmax loss'.
- line 127: This is pure speculation. Please remove or provide an argument.
- line 168: I recommend that you explain more clearly how your proposed metric relates to [43]. Is it a similar methodology of LCA levels, just applied to a different base metric? something entirely different?
- 19 classes does not seem particularly long-tailed compared to e.g. class distributions in Mapillary Vistas. I understand the argument that the rare classes are tail-classes because they are orders of magnitude less frequent than the majority classes. However, I think it would help to quantify and relate this: How many orders of magnitude are between tail classes and majority classes in other datasets? Are results from the nuScene dataset with only 19 classes relevant to long-tailed classification in general?
- lines 216ff: Please discuss your training approach in relation to other works. Why did you follow this approach? How does it relate to existing hierarchical classification approaches? What is novel about your approach?
- line 216: Do you also modify the RGB-based detectors?
- Table 1: Is '+Camera' for TransFusion a typo? If not, what does it mean? Do you not train TransFusion with hierarchies?
- lines 240-241: Why did you downsample instead of using crops during training? Does a single image for inference not fit into memory? Also, please indicate in all tables that TransFusion has less information available (downsampling makes detection of far-away objects harder) compared to your multimodal filtering approach.
- Table 2: Can you please add here a comparison with a detector just trained on the coarse training protocol from nuScenes (basically trained directly only on the LCA=1 level)? The same could be done with LCA=2. Both would give a good indication of what established training methods would achieve for this evaluation level, and give context what a potential 'upper bound' is.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper studies the Long-Tailed 3D Detection (LT3D) problem in the application domain of autonomous vehicles (AVs). It argues that accurately detecting long-tailed classes in AV datasets is crucial to their safe and effective deployment in real-world applications. Most current benchmarks, however, either: (a) do not measure the performance of state-of-the-art detectors on these long-tailed classes, or (b) group them with other, more commonly occurring classes, inhibiting detector performance on these classes. The paper addresses these concerns by proposing architectural improvements, such as training with semantic hierarchies and multi-modal filtering. It also presents a hierarchical evaluation metric for measuring the performance of a detector on all classes, including long-tailed classes. The experimental evaluation evaluates different detectors on the nuScenes dataset according to the proposed metric.
strengths_and_weaknesses: On the positive side, the paper is well-written and the problem of LT3D is clearly defined. The experimental section is thorough in terms of the detection methods being evaluated. It evaluates both multi-modal and non-multi-modal detectors, which is appreciated. It also presents an ablation study of the different detector components proposed in Section 3. Table 1 summarizes the state-of-the-art in this problem domain and highlight the key insights from different detection methods. It presents clearly the technical improvements proposed to state-of-the-art detectors and the evaluation metrics designed to measure LT3D performance.  The survey of related work also appears to be representative of some of the related work.

In terms of limitations, the main concern is that there has been a lot of work in long-tailed recognition, especially for the 2D case, with similar ideas to those proposed in the current paper on how to address this challenge (e.g., hierarchical reasoning). Some of these efforts are mentioned in Section 2. The paper does not do a good job at this point, however, of highlighting how the current work differentiates from them beyond trying the same ideas in the 3D domain.  Stronger justification beyond “...rare classes are not only infrequent, but are also difficult to distinguish using LiDAR alone” would be helpful to emphasize the importance of the challenge in this domain. Because the proposed solution appears to be rather standard given the literature and it is not clear how the nature of the LiDAR data affects the approach. 

While there is some improvement in detection using hierarchies and multimodal filtering (Table 2), it does not reach the point of sufficiently supporting the claim that these solutions are addressing the challenge. For instance, the mAP_H values for the "Child" and "Pushable-Pullable" classes are still rather low.

Furthermore, the experimental evaluation only considers a single dataset, nuScenes. The argument made in Section 5.1 is that nuScenes explicitly defines the semantic hierarchies required for the evaluation protocol proposed in the paper. Nevertheless, it should be possible to use external semantic hierarchies (e.g., WordNet) for other dataset labels.

In Section 3, Line 110, it is stated that "...heads of rare classes overfit..". Then, in Section 3, Line 120, "...use a per-class logit-loss rather than multi-class softmax loss...". Why does the latter not suffer from the same overfitting problem for rare classes.  

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper is well written and the technical direction for approaching the problem of LT3D is sound, while there are accompanying experiments that use state-of-the-art detectors. The novelty, however, is not clear given the amount of work in long-tailed detection and the manuscript does not sufficiently differentiate the challenge from the work in 2D detection. The arguments would also be better supported if the proposed changes were evaluated on more than a single dataset, where it may be possible to argue stronger improvements.
issues: Better support the novelty and significance of the work in context of prior and ongoing work in the area of computer vision algorithms focusing on long-tailed detection.

Experiments on additional datasets. The semantic hierarchy can be manually handcrafted. It would be acceptable to evaluate only the CenterPoint and Transfusion detectors.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper studies a very important problem in autonomous driving: long-tail 3D object detection. The authors propose several useful training methods and did an in-depth evaluation, which gives the readers some valuable insights.
strengths_and_weaknesses: ### Strengths:

1. The in-depth analysis in both the method and experiment section provides the readers some very useful insights on training 3D object detectors in long-tail classes, such as using group-free detection heads and training with semantics hierarchies without softmax classification. 

2. The question “Are objects from rare classes sufficiently localized but mis-classified” is very valuable. The authors claimed and showed that the LiDAR detector can reach a high recall in terms of 3D localization, but poor at classification. Then the authors use RGB modality to filter out the false positives, and improve the classification performance. 

### Weaknesses:

1. The paper does not propose a significantly novel framework. But the analysis and insights are interesting.

In summary, the paper provides a valuable analysis on training 3D object detectors for long-tail classes. I recommend for publication because this paper could potentially inspire new methods.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Recommend for publication
issues: What if the lidar detector gives many false positives along a single line of sight on the RGB image? How can the RGB-based detector classifiy which one is the most correct?
review_title: null

Meta-review: metareview: The reviewers all appreciate the relevance of the long-tailed 3D object detection, and the proposed evaluation metric used to also account for rare classes is very useful. Also, the proposed ideas of multi-modal filtering and hierarchical classification are very helpful to improve the classification performance on rare classes. On the downside, the reviewers argue that there are too little implementation details given in the paper (e.g. what is the used loss function?), and an experimental comparison to other hierarchical classification methods (which are given in the related works) is also missing. Some further discussion on these points in particular would therefore be very helpful during the rebuttal.  

Post-rebuttal:
The authors have given detailed answers to most of the points raised by the reviewers. The additional experiments and evaluations give a more detailed insight into the performance of the proposed method, in particular the comparison to other hierarchy-based techniques. It would be good to move at least part of this discussion from the supplementary into the main text. Also, in Table 2 highlighting the best numbers in bold would help readability. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)