Learning 3D Perception from Others' Predictions

Authors: Jinsu Yoo, Zhenyang Feng, Tai-Yu Pan, Yihong Sun, Cheng Perng Phoo, Xiangyu Chen, Mark Campbell, Kilian Q Weinberger, Bharath Hariharan, Wei-Lun Chao
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D object detection, autonomous driving, label-efficient learning, domain adaptation, curriculum learning, collaborative perception

Abstract:
Accurate 3D object detection in real-world environments requires a huge amount of annotated data with high quality. Acquiring such data is tedious and expensive, and often needs repeated effort when a new sensor is adopted or when the detector is deployed in a new environment. We investigate a new scenario to construct 3D object detectors: *learning from the predictions of a nearby unit that is equipped with an accurate detector.* For example, when a self-driving car enters a new area, it may learn from other traffic participants whose detectors have been optimized for that area. This setting is label-efficient, sensor-agnostic, and communication-efficient: nearby units only need to share the predictions with the ego agent (e.g., car). Naively using the received predictions as ground-truths to train the detector for the ego car, however, leads to inferior performance. We systematically study the problem and identify viewpoint mismatches and mislocalization (due to synchronization and GPS errors) as the main causes, which unavoidably result in false positives, false negatives, and inaccurate pseudo labels. We propose a distance-based curriculum, first learning from closer units with similar viewpoints and subsequently improving the quality of other units' predictions via self-training. We further demonstrate that an effective pseudo label refinement module can be trained with a handful of annotated data, largely reducing the data quantity necessary to train an object detector. We validate our approach on the recently released real-world collaborative driving dataset, using reference cars' predictions as pseudo labels for the ego car. Extensive experiments including several scenarios (e.g., different sensors, detectors, and domains) demonstrate the effectiveness of our approach toward label-efficient learning of 3D perception from other units' predictions.

Decision: accept (poster)
Reviewer scores: 5, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729578112161

thread_type:
official_review

<thread_object>
creator: reviewer_djew

<content>
summary:
This paper designs a new strategy to train the 3D object detector, i.e. learning from the detection results of a nearby unit equipped with a  pretrained detector. There are two main challenges, mis-localization and viewpoint mismatching, resulting in the inferior performance of naive self-training. To this end, this paper proposes a rank-based pseudo-label refinement module and a distance-based curriculum learning scheme. Experiments are conducted on V2V4Real and OPV2V datasets to evaluate the proposed algorithm.

strengths:
1. This paper proposes a creative and interesting strategy for the training of object detectors. It can exploit existing pretrained detector in a flexible way regardless of their input formats and deployment compatibility. The authors provide detailed explanation about their motivation and sound justification of the feasibility.

2. The algorithm includes a relatively complex pipeline. However, each module inside it is well-motivated and has a clear role. The step-by-step improvement brought by each module is also demonstrated clearly through extensive experiments.

3. The paper is well-written. The authors organize everything well to make it easy to follow. The illustrations are also very clear, helpful for  readers to understand the content.

weaknesses:
1. The authors are targeted for LiDAR sensors. However, if we apply this setting in practice, the camera sensors are much more ubiquitous such as the surveillance. I am wondering whether the algorithm would still work if the reference car is using camera-based detector. Given camera-based detectors are weaker, this may require high robustness of the proposed algorithm.

2. In the experiments, it may be helpful to include some baselines using other strategies for the training of detector. For example, since the algorithm requires a small number of annotated frames to train the ranker, it may alternatively apply some semi-supervised strategies to train the detector directly.

questions:
Please consider replying to the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730396264429

thread_type:
official_review

<thread_object>
creator: reviewer_goe2

<content>
summary:
This papers proposes the novel idea of "learning from others predictions", i.e. using pseudo-labels from a detector deployed on another agent in the scene to train an ego agent perception model. The approach can be summarized in three parts: first, the reference detector is pre-trained on a human-annotated labels offline, and a box ranker is pre-trained on a small number of human-annotated ego frames. Next, the reference detector's pseudo-labels are generated for frames in which the reference agent is below some threshold distance from the ego agent, and are used to train the ego detector after undergoing a course-to-fine box refinement module with the box ranker. Lastly, self-training is performed using the ego detector's predictions on all available frames. Their approach, dubbed R&B-POP, is able to significantly improve AP from naively using the reference detector predictions, nearly equaling performance of training with human-annotated labels on V2V4Real Collaborative Perception dataset.

strengths:
1. The setting introduced in this paper is novel and creative, and will inspire future work in this new research direction.
2. The method is well-explained and intuitive. Using a set of thoughtful approaches for improving psuedo-label quality and training, the authors are able to significantly improve performance over the baseline and other heuristic approaches.
3. Experiments are very thorough, and many ablation studies have been performed. 
4. The paper is well-written and organized.

weaknesses:
1. Although the results are impressive on V2V4Real, the experiments are performed on the case where there is minimal domain shift between the two vehicle detectors (e.g. they both have the same sensors, both survey the scene from the street, etc). It would be a more interesting experimental setting with two drastically different object detectors, e.g. one on a LiDAR mounted car, and one on a stationary security camera. The authors suggest scenarios such as these as possible applications, although it's not clear that the method demonstrated in this work can be easily applied to such a scenario. However, I understand experiments are limited by available data, so these may not be feasible in practice.
2. The authors utilize a small number of human-annotated frames from the Ego vehicle sensor data to train the box ranker, so the need for a human labeler is not completely eliminated. It would also be compelling to include experiments comparing to other semi-supervised learning approaches for 3D object detection using a similar number of labeled ego frames, to demonstrate that R&B-POP works better than conventional SSL approaches for this level of annotator budget.

questions:
Have the authors tried training the ranker without using any annotated ego frames? As mentioned previously, it would be compelling if the need to label any of the ego data was completely eliminated.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730544444493

thread_type:
official_review

<thread_object>
creator: reviewer_lf3v

<content>
summary:
The paper investigates a new scenario for constructing 3D object detectors in autonomous driving by leveraging predictions from nearby units equipped with accurate detectors. The paper addresses the challenges of viewpoint mismatches and mislocalization due to synchronization and GPS errors. The authors propose a learning pipeline called Refining & Discovering Boxes for 3D Perception from Others' Predictions (R&B-POP). This pipeline includes a distance-based curriculum that starts by learning from closer units with similar viewpoints and then improves the quality of predictions from other units through self-training. In general, while there are no surprising design elements in the methodology, the problem addressed in the paper is a new issue in this field. Learning new knowledge from the predictions of other agents appears to have significant practical application value.

strengths:
1. The paper is well-structured, with a clear abstract, introduction, methodology, experiments, and conclusion sections that logically flow from one to the next.
2. The proposed  Refining & Discovering Boxes for 3D Perception from Others’ Predictions (R&B-POP) method is simple and effective.
3. The paper has extensive experiments and the figures are very professionally made.

weaknesses:
1. The scenario assumed in the paper has great limitations, that is, learning knowledge from nearby agents to achieve detection. Theoretically, the learning process needs to be fast enough so that the current vehicles can predict new objects. However, I did not see the authors' analysis of the algorithm's training speed and discussion of actual deployment limitations.
2. Box ranker seems to be just an IoU scoring method, which is adopted in most detectors, such as Voxel-RCNN, PV-RCNN, they are not fundamentally different, why not directly use or discuss the limitations of existing IoU scoring method in this application?

questions:
1. It is possible that the method can learn knowledge from other vehicles in real-time, which would expand the scope of practical applications?
2. What's the key difference between Box ranker and IoU scoring in traditional detectors?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730567293747

thread_type:
official_review

<thread_object>
creator: reviewer_iyjb

<content>
summary:
This paper aims to develop a label-efficient approach for training 3D object detectors in autonomous vehicles by leveraging predictions from nearby vehicles equipped with accurate detectors. The main technical contribution is the proposed R&B-POP pipeline, which refines detection results from neighboring cars using a box ranker and a distance-based curriculum self-training approach. This method addresses challenges like viewpoint mismatches and localization errors that arise when integrating predictions from external sources. In experiments, the authors validated their approach on a collaborative driving dataset; they showed that with a small set of labeled data, this approach achieves performance close to that of fully supervised models.

This is an interesting exploration of new formulation for label-efficient 3D object detection. However, it is unclear how well the proposed method performs compared to other data-efficient domain adaptation or fine-tuning strategies, as relevant comparisons with SOTA are lacking. Additionally, the problem setting feels somewhat artificial, relying on several assumptions that may limit its practical applicability. 
My rating is a weak reject.

strengths:
+ The paper introduces an interesting problem formulation for 3D object detection in resource-limited settings, suggesting a new collaborative information-sharing approach to reduce labeling costs.
+ The ablation study is well designed. Results demonstrate the contribution of different design components to the overall performance, sharing valuable insights into the effectiveness of these different components in the proposed pipeline.

weaknesses:
-	Practicality and applicability of the problem setting. One of my main concerns is the practicality of the proposed problem setting. First, the approach relies heavily on the assumption that neighboring cars with accurate detectors are always available when needed, but this may not always be feasible. Additionally, without having a good pre-trained detector but attempting to rely on other cars’ detection sharing seems unsafe. This mechanism will also need to resolve multiple other dependencies and complexities such as identifying trustworthy vehicles (e.g., not malicious, not inaccurate), managing potential error accumulation (causing problems when passing to other cars), etc. Without all these sorted out, the current problem setting is somewhat artificial and lacks reliability and scalability. 
-	Technical comparisons. If the primary technical goal is to enable fine-tuning with minimal data, it should include comparisons with existing/ SOTA data-efficient training and fine-tuning methods. Demonstrating how this method compares with these techniques/models, will better clarify its advantages and limitations. Such a comparison is currently lacking.   
-	Another minor issue: when comparing with the fully supervised models, please also include comparisons with SOTA models. Additionally, consider testing how the accuracy changes if fewer data are selected/used for training. This comparison would better illustrate the proposed pipeline’s advantage over simply having a road-side transmitter providing local data to passing vehicles.

questions:
please see my major concerns in the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730635714741

thread_type:
official_review

<thread_object>
creator: reviewer_r3jy

<content>
summary:
This paper introduces a novel approach for training 3D object detectors by utilizing predictions from nearby agents with accurate detectors. This approach addresses the challenges of acquiring large amounts of annotated data. Validated on a real-world collaborative driving dataset, the proposed method demonstrates effective label-efficient learning and substantially improves the AP.

strengths:
- The quality of the paper's figures is high and relatively clear and explicit.
- This method ensures the accuracy of the detector while reducing the cost of annotation, and its effectiveness has been verified on both real and simulated datasets.

weaknesses:
1. This paper introduces a variant of the offboard 3D object detection method. It only selects an unsupervised scheme as a baseline, which is inappropriate. For a fair comparison, the proposed paper should be compared with offboard 3D object detection methods with similar experimental settings. These methods also rely on an accurate detector to label unannotated scenes. The authors should select the correct baselines[1] [2] for comparison. 
[1] DetZero: Rethinking Offboard 3D Object Detection with Long-term Sequential Point Clouds, ICCV 2023. 
[2] Offboard 3D Object Detection from Point Cloud Sequences. CVPR 2021.

2. Compared to traditional offboard 3D object detection, the proposed solution not only uses full data annotation to train an accurate detector but also provides partial manual labeling for unannotated scenarios. This approach of adding extra manual labeling diminishes the contribution of the method. The authors emphasize that when an autonomous vehicle enters a new scene, it can obtain scene information from an established intelligent agent, which is an online process. However, the method itself is offline, which creates a contradiction. What are the proposed method's contributions in light of these concerns?
3. The proposed method provides pseudo-labels for the ego vehicle by directly passing the predicted results of the reference vehicle, which is a very naive approach. A semi-supervised learning framework is more flexible. What kind of performance effects would a semi-supervised learning framework have in this experimental setup? For example, train a teacher network using data from labeled reference vehicles, and then use the teacher network to generate pseudo-labels for unlabeled point clouds to train the student network further.
4. The ground truth of the reference vehicle is completely discarded in the subsequent method design, which is undoubtedly regrettable. Even if the center position of the ground truth has a deviation for the ego vehicle, the size of the bounding box is entirely correct, and this information is also very important. Therefore, why not use the reference vehicle's ground truth in their method?

questions:
The proposed method provides pseudo-labels for the ego vehicle by directly passing the predicted results of the reference vehicle, which is a very naive approach. A semi-supervised learning framework is more flexible. What kind of performance effects would a semi-supervised learning framework have in this experimental setup? The ground truth of the reference vehicle is completely discarded in the subsequent method design, which is undoubtedly regrettable. Even if the center position of the ground truth has a deviation for the ego vehicle, the size of the bounding box is entirely correct, and this information is also very important. Please also respond to the comments in  Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732141271422

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We thank all reviewers for their valuable comments. We particularly appreciate that the reviewers recognized that our proposed scenario is interesting, novel, and creative (iYjB, goE2, dJeW); our method is well-motivated, well-explained, and intuitive (R3jy, LF3v, goE2, dJeW); our experiments and analyses are thorough and well-designed (R3jy, iYjB, LF3v, goE2, dJeW). We humbly think that these are essential for introducing a new scenario that the community can follow and further improve. Below, we have posted our initial responses to each reviewer to address most of the concerns, if not all. We would appreciate the reviewers’ timely feedback so we can further discuss it.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732729233850

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely appreciate the reviewers’ thoughtful and constructive feedback. Based on your suggestions, we have made improvements to our manuscript. The updated text has been highlighted in red. A summary of the changes is as follows:

- Added new references on offboard and semi-supervised methods in Related Work. (**R3jy**)
- Included a discussion on offboard methods in Section S1. (**R3jy**)
- Expanded the problem definition and setup in Section S1 to provide more clarity. (**R3jy**, **LF3v**)
- Justified the ranker design in Section S2 under the paragraph on ranker architecture. (**LF3v**)
- Provided additional explanation about the experiment with the ranker without any human labels in Lines 469 - 470. (**R3jy**, **goE2**)
- Added experimental results (Table S1) describing the impact of training data size on detector performance in Section S3.1. (**iYjB**)

We are also conducting further experiments to compare our approach with a semi-supervised method, as suggested by the reviewers. **These additional results are in progress, and we anticipate sharing them soon.** We appreciate your understanding and patience in waiting for this analysis. 

Thank you again for your valuable feedback, which has greatly strengthened our work!
</content>
</thread_object>

</thread>
<thread>
creation date:
1733286488800

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Additional general response [1/2]

comment:
Dear reviewers,

We appreciate all the reviewers’ review comments and interactions during the discussion period. We provide further general responses as follows.

### Comparison to label-efficient, semi-supervised learning methods

We apologize for the delay in providing the requested results investigating other semi-supervised learning methods (R3jy, goE2, dJeW). In the past week, we suffered an unexpected server shutdown. Fortunately, we managed to have the results ready, as follows.

To begin with, let us reiterate our main contribution, which is to explore a novel scenario of using the reference car’s predictions as supervision to train the ego car’s detector. In our main results, we annotated 40 frames of the ego car’s data to train the ranker to refine the reference car’s predictions (Line 286). The refined predictions were then used as pseudo labels to train the object detector for the ego car (Section 3.3). We note that our approach already involves semi-supervised learning, using self-training to fill in missing labels and, in turn, re-train the model (Section 3.4). **We have shown that it is feasible to bypass the 40 annotated frames but use synthetic data to train the ranker** (Table 3a and Lines 469 - 470; see also responses to R3jy).

As reviewers R3jy, goE2, and dJeW suggested, we have investigated a direct semi-supervised approach, using the 40 annotated frames and other unannotated frames of the ego car’s data to train the detector. We apply 3DIoUMatch [a], a widely used and representative semi-supervised learning approach in this setting. We note that the official code of 3DIoUMatch used the PV-RCNN detector, not the PointPillar detector in our main paper. As such, we rerun our approach using the PV-RCNN detector for a fair comparison.

*Table: Additional experiments on semi-supervised approach. The performance is reported on AP at IoU 0.7. * and $\dagger$ indicate supervision and approach, respectively.*
| labeled 40 frames$^{*}$ | _R_'s pred$^{*}$ | 3DIoUMatch$^{\dagger}$ [a] | Ranker$^{\dagger}$ (Sec 3.3) | Curriculum$^{\dagger}$ (Sec 3.4) | 0-30m | 30-50m | 50-80m | 0-80m |
|:-----------------:|:--------:|:--------------:|:----------------:|:--------------------:|:-----:|:------:|:------:|:-----:|
|         o         |          |        o       |                  |                      |  60.7 |  24.4  |   6.0  |  37.1 |
|         o         |     o    |                |         o        |           o          |  65.1 |  34.1  |  12.6  |  41.7 |
|         o         |     o    |        o       |         o        |                      |  60.6 |  30.7  |   8.6  |  39.5 |
|         o         |     o    |        o       |         o        |           o          |  68.1 |  36.8  |  12.9  |  44.4 |


The above table summarizes the results. Row 1 is the result of 3DIoUMatch [a], and Row 2 is the result of our approach (cf. Row 10 in Table 2, but using PV-RCNN as the detector). We see that our approach outperforms 3DIoUMatch, demonstrating the value of using reference cars’ predictions as auxiliary supervisions. 

That said, we certainly do not claim that our approach is a better semi-supervised learning approach than 3DIoUMatch (or other direct semi-supervised methods). First, we leverage reference cars’ predictions. Second, ours and 3DIoUMatch use the 40 annotated frames in drastically different ways—ours uses them for ranker training, while 3DIoUMatch uses them for detector training. As such, they are potentially complementary to each other. If integrated appropriately, they may lead to even better results. 

To investigate this idea, we use our ranker to refine reference cars’ predictions and add those high-quality ones (_i.e._, < 40 meters, defined in Section 3.4) as extra labels to 3DIoUMatch. Row 3 shows the results: we see a 2.4% boost in 0-80 meters against Row 1, justifying the compatibility of ours and 3DIoUMatch. On top of Row 3, we further apply our distance-based curriculum for self-training (Section 3.4), using 3DIoUMatch’s predictions on all the data as pseudo labels to re-train the detector. Row 4 shows the results: we see another 4.9% boost against Row 3 and 2.7% boost against Row 2. In sum, these results demonstrate 1) the effectiveness of our approach in leveraging reference cars’ predictions as supervision (Row 3 and Row 2 vs. Row 1) and 2) the compatibility of our approach with existing direct semi-supervised learning approaches to further boost the accuracy (Row 4 vs. Row 3 and Row 2). We view such compatibility as a notable strength: it demonstrates our approach as a valuable add-on when nearby agents’ predictions are available. 

Once again, we appreciate the reviewers pointing out an important topic to discuss. We will include these findings in our final version.

---
[a] Wang et al., "3dioumatch: Leveraging iou prediction for semi-supervised 3d object detection," in CVPR 2021.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper received mixed reviews, with four reviewers voting for borderline acceptance and one reviewer (R3jy) voting for borderline rejection. The key contributions of this paper include the introduction of a new label-efficient setting for 3D object detection, which leverages prediction results from nearby agents, and the design of the first solution to address this novel setting.

After the rebuttal, Reviewers LF3v, goE2, and dJeW acknowledged that most of their concerns had been addressed, while Reviewer iYjB did not respond, and Reviewer R3jy actively engaged in the discussion with the authors. Since Reviewer iYjB also did not participate in the AC-reviewer discussion period, the AC reviewed the authors' response to their questions and believes that most concerns were resolved. 

The remaining concerns from reviewers focused on insufficient comparisons with label-efficient methods (R3jy, dJeW) and offboard methods (R3jy). The AC noted that neither Reviewer R3jy nor dJeW responded to the authors' latest reply. After carefully reviewing the authors' follow-up responses, the AC agrees with the authors' argument that offboard 3D object detection does not constitute a directly comparable setting to the proposed method, even though both aim to generate pseudo labels from existing models. Regarding the comparison with other label-efficient methods, such as semi-supervised methods, the authors addressed this concern by providing a comparison with 3DIoUMatch and demonstrating a performance improvement through the incorporation of a semi-supervised technique.

As a result, the AC believes that the concerns raised by Reviewer R3jy have largely been addressed. Therefore, the AC recommends accepting the paper and encourages the authors to include the newly added experimental results in the final version.