Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes Rote-DA, a method for unsupervised domain adaptation for 3D detectors.  
By using LIDAR readings over multiple traversals in the target domain,  
they use the "persistence prior score" (PP-score, prior work) to filter out false-positives and provide additional supervision for reducing false negatives.
They conduct experiments on the Ithica-365 and Lyft datasets and show their adaptation strategy improves adapted detector performance.
strengths_and_weaknesses: ### Strengths
* Written clearly, technical portion straightforward and well inspired
* Simple and improved performance

### Concerns
* Only provides comparisons to two other methods
* Method only shown on PointRCNN
* Slightly worried about technical novelty - FB-F and FB-S could be seen as just a few filtering tricks
* Applicable to only dynamic objects (doesn't work for road divider, traffic cone, street lights/signs)
questions: * Is there any easy way to adapt this method to handle road divider / traffic cones / street lights / signs?
* It seems the self-training only improves the cyclist class - can the authors give any insight on this?
* Have there been any experiments with other architectures? Mostly curious about single-stage detectors.
* How many traversals of the same scene are in the datasets?

### Miscellaneous
* Technical portion is well written but a figure summarizing the filtering would be nice to look at
* The method is motivated by the fact that repeated traversals will be collected by end users, but for now most (if not all) cars do not have lidar sensors.
* FYI - the nuScenes dataset has repeated traversals
limitations: yes
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper tackles the task of 3D object detection in LiDAR point clouds, particularly detecting moving objects such as cars, pedestrians, and cyclists, with a strong emphasis on applications for autonomous ground vehicles. The authors leverage past traversals from the target domain, in order to refine pseudo-labels (using 3 refinement procedures: PO-F, FB-F, and FB-S) and adapt a pre-trained (on the source domain) PointRCNN model in a bid to improve generalization without the effort of novel annotations (by refining predictions on unlabeled data). 
strengths_and_weaknesses: Strengths:
* Originality: similar work has been done before, but I am not aware of any pseudo-label refinement procedures in LiDAR point clouds using the persistency scores
* Quality: the paper is well-written and well-structured, all details needed for reproducibility have been provided
* Clarity: Good to see the full thought process for each of the steps involved in the method (backed up by ablative studies - Sec. 4.2) although there are some aspects in the evaluation procedure that create confusion - e.g. L242-243 vs. L244-245 conflicting evaluation procedures? Is it a frontal view or BEV evaluation?

Weaknesses:
* Significance: Some arguments are not backed up by proper experimental analysis - e.g. L184 - the values for those thresholds are hard to swallow without proper analysis - these were validated on which dataset? Do they apply for both Lyft and Ithaca or is validation done on KITTI? Also, the experimental setup is limited to only one source domain (KITTI) compared to previous work e.g ST3D which varied the source and target domains.

Minor comments:
* Figure 1 is placed on page 3 and it's first presented on page 5 - maybe it should be placed on the page closer to the text reference 
* L103 - Remove "to" - "... uses the signal to _to_ learn"
* L111-112 - complementary work to very recently published work which is a plus
* Table 3 caption - use "two" instead of "to" in "... can be further subdivided into _two_ subcomponents"
* Table 3 - which testing scenario did you report the numbers on - failed to find this information in the text as well.
* L191 - rephrase - remove first "is" or the second, but keeping both makes no sense in "... One naive way is to remove them _is_ by thresholding"
questions: * How does your work compare to Hindsight (reference [36] in the paper)? Based on the reported numbers on Lyft they perform better, but I did not check carefully if the evaluation scenario is similar. Is it feasible to do a fair numerical comparison between your method and theirs? 

* Regarding statement from L93-94 - Do the authors have any experiments to back up this theory? How do the authors know the gains would be marginal? Numerical arguments or at least a more thorough argument. 

* Regarding the statement from L224-226 - What about the nuScenes dataset [1]? Why wasn't it included in the list? Hindsight uses past traversals and reports numbers on nuScenes.
[1] - Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., ... & Beijbom, O. (2020). nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 11621-11631).

* The numbers using ST3D look terrible, especially for R30. The authors should state in the paper if this is a reimplementation and replicate one of the experiments reported in the ST3D paper to make results credible (which could be an easy task since ST3D uses 4 datasets: Waymo, KITTI, Lyft, and nuScenes). And again my question regarding the limitation to using just KITTI as a source domain? At least something comparable to what is presented in ST3D - "(1) adaptation from label rich domains to label insufficient domains (i.e., Waymo to other datasets) and (2) across domains with different numbers of the LiDAR beams (i.e., Waymo → nuScenes and nuScenes → KITTI)"
limitations: Privacy concerns properly addressed in the "Conclusion" section, but I did not find a dedicated section addressing the current limitations of the proposed method (I think this could be very useful). Some of the initial assumptions could be considered limitations (e.g. (1) the fact that this method works only if the objects of interest are dynamic, (2) the unlabeled data from the target domain are collected from the same routes at a relatively close period of time and (3) the source and target domains are not from drastically different areas (city vs. farmland)).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper performs iterative self-training of 3D object detectors on the target domain. Specifically, the authors assume that multiple traversals are particularly suited for end-user domain adaptation, and leverage persistency prior (PP-score) to provide a powerful signal to both false positives and false negatives from detector outputs. In this way, the detector can generate the filtered pseudo labels. This paper also introduces a new auxiliary loss that forces the detector to classify non-persistent LiDAR points as foreground.
strengths_and_weaknesses: Strengths:
1. This paper solves end-user domain adaptation by the self-training car setting that gives rise to a weak supervision signal that is exceptionally well-suited to adapting 3D object detectors to a new environment.
2. The paper leverages unlabeled LiDAR data that automatically comes in the form of multiple traversals of the same routes to iteratively refine a detector to new domains.
3. The authors propose Posterior Filtering, Foreground Background Filtering filtering, and Foreground Background Supervision to correct and reduce both false positives and false negatives.

Weaknesses:
1. It seems that Rote-DA cannot perform well in class Car at IoU 0.7/0.5 on KITTI -> Lyft adaptation.
2. There are many hyper-parameters, e.g., alpha, beta.
questions: 1. In Table S4 in Supplementary, why Rote-DA cannot perform well in class Car with IoU 0.5.
2. How to choose suitable hyper-parameters?
limitations: This paper proposes effective methods (i.e., posterior filtering, foreground background filtering, and foreground background supervision) to solve end-user domain adaptation with unlabeled data. There are a few limitations that this method cannot perform well in class Car at IoU 0.7/0.5 on KITTI -> Lyft adaptation, and has several hyper-parameters needed to be tuned.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on the task of unsupervised domain adaptation (UDA) for object
detection in 3D point clouds by automatically inferring additional labels in the
otherwise unlabeled target domain.

The method works best with two-stage detectors, in which case it can be applied
without modifying the model. The method is somewhat orthogonal to other UDA
techniques such as Waymo's SPG or any of the simulator-based domain adaptation
techniques, though evaluating such combinations is beyond the scope of the
paper.

The goal is to adapt a detector trained on a source environment to a new target
environment where labels are unavailable. This is done by leveraging information
from multiple passes covering the target domain. Specifically, the LiDAR
information from these passes, assumed to be co-registered, is used to compute
statistics about the likelihood that a given point belongs to a dynamic
foreground object.

These point-wise statistics are used in two ways:
 - (1) to refine the detector outputs in order to produce pseudo-labels by
   refining its predictions to remove false positives;
 - (2) as weak supervision for the proposal stage of the detector;

This process can be iterated multiple times. Predict, "clean up" predictions to
improve pseudo-labels, re-train, repeat. 

The evaluation is thorough, with experiments demonstrating the generalization of
a Point-RCNN detector trained on KITTI, showcasing promising results on both the
Lyft, as well as the Ithaca-365 datasets, outperforming a couple of recent
baseline UDA methods.

strengths_and_weaknesses: ## Strengths
- [S1] The paper is well-written and clear, and its topic is highly relevant to
  the autonomous driving industry.
- [S2] Detailed analysis and promising results in UDA, outperforming several baselines.
- [S3] The fact that the authors plan to release the code is very helpful and would
  be a great contribution to the community.
- [S4] The labels produced by this approach can be post-processed into tracklets and
  even used to train more complex tasks such as multi-sweep detection,
  online tracking, and motion forecasting. As a suggestion, it would actually be
  nice to mention this as further possible applications of the paper, as this is
  a very important benefit of a label-oriented domain adaptation technique.

## Weaknesses
- [W1] *Single-stage vs. two-stage*: I would make it a bit clearer that a
  significant portion of the method's benefits come from the FB-S, which, in
  turn, relies on the underlying detector being two-stage. I don't doubt that
  this method can also be applied to single-stage detectors(*), but it would
  clearly require non-trivial modifications to work with SSDs.
  - (*) SSDs are still preferred in many applications because of their low
    latency, for example.
- [W2] (Minor) The improvement is only demonstrated on a single, relatively old
  LiDAR object detector (Point R-CNN). I don't think it's a dealbreaker but if I
  were to pick one thing to strengthen the paper it would be evaluating on more
  than one detector; I know OpenPCDet includes several newer detectors like
  PV-RCNN++, so it would be nice to see how those work in the proposed
  framework. 

## Suggestions
- In the "Repeated Traversals" related work section, one "oldschool" area
  which has been using multiple passes for a really long time is mapping. For
  instance [levinson2010robust] use multiple passes to build a LiDAR
  intensity map, which they represent probabilistically by modeling the
  intensity of each map cell as a Gaussian. The reflectivity variance thus
  acts as a measure of the reliability of each cell for localization. While
  different from the task presented in this paper, it would be nice to mention
  this classic line of robotics work.
- L191: "way more numerous" or just "more numerous"?
- L315: "filtering" is repeated
- Citation [16] Has incorrect capitalization on one of the names.

## References
- [levinson2010robust]: Levinson, Jesse, and Sebastian Thrun. "Robust vehicle
  localization in urban environments using probabilistic maps." 2010 IEEE
  international conference on robotics and automation. IEEE, 2010. 

questions: - [Q1] How much does PO-F still help when FB-F and FB-S are enabled? (The
  one missing line from Table 3, basically.)
- [Q2] Is the PO-F cap applied per-frame or per-dataset? I.e., do you do the
  rank-and-select within each scene, or for the whole target dataset? (My
  understanding is that you do the latter but I just wanted to double-check.)
limitations: Please see "Weaknesses" and "Questions".
review_title: null

Meta-review: metareview: This paper proposes to leverage unlabelled LIDAR scans from vehicles repeatedly traversing the same environment (e.g., Lyft data in the US), for the domain adaptation of car, bicycle and pedestrian classifiers trained in different domains (e.g., KITTI data in Germany). It uses pretrained Point R-CNN classifiers and bounding box detectors followed by Persistent Point PP-scores and statistical methods for false positive and false negative removal. Various ablations demonstrate the superiority of this self-training method.

Reviewers had very split scores, going from 8 (UWLf) down to 3 (​​mdTS). After rebuttals, reviewer's estimated score is 5 (not updated in the review, but the reviewer said they were "favourable"), meaning an average score of 5.75. 

Reviewers praised the idea (UWLf, YZhW, mdTS), the clarity of the paper (UWLf, FkR5, mdTS), the thorough evaluation (UWLf, mdTS), the promising generalisation results on the Lyft and Ithaca-365 datasets (UWLf, YZhW, FkR5) and the potential uses of unsupervised data labeling (UWLf).

Reviewer (UWLf, FkR5) noted that other LIDAR point classifiers other than Point R-CNN or single-stage detectors could have been evaluated: these points were addressed during rebuttals with new experiments). Reviewer (YZhW) noted poor performance in one specific domain and the presence of many hyperparameters, but did not respond to the authors’ rebuttal. Reviewer (FkR5) was concerned about applicability to static objects, limited novelty of the filtering methods, and the fact that comparison was  with only two other methods; the authors provided a rebuttal to most of these points. Reviewer mdTS had a large number of specific questions regarding the evaluation, starting with the hyperparameter choice; again, the reviewer did not answer to the authors’ rebuttal.

As AC, and based on scores 8, 5, 5, and 3->5?, I would recommend this paper for acceptance.

Sincerely,
Area Chair
confidence: Less certain
Decision: Accept