Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes the use of playbacks for UDA. The author uses the trained model and an offline 3D object tracker to generate high-quality pseudo-labels of the target domain. After that, the original model is fine-tuned on the generated pseudo-labels to improve performance on the target domain.

The paper can be understood in general and the writing is easy to follow. The results of the paper are practical. It's reasonable because it can generate more accurate 3D boxes for the target domain, especially for those long-distance objects. The authors have done experiments on 5 data sets to show the generalization capability of the method and compare the two decent baselines with the proposed method. 

With the video of the same scene over time, I am also curious if the effect of using the point cloud of the previous/next frames to enhance the point cloud data of the current frame, which may further enhance the effect and generalization ability of pseudo-labels.

The major novelty of the paper is the combination of offline-tracking and self-training techniques, which is practical for real-world engineering problems. However, in general, I think the novelty is still limited to the ICLR community. In my view, the only difference between the proposed method and ST is the introduction of video information (an assumption) and the offline tracker to make the pseudo-label more accurate.
review_title: Practical but lack of novelty

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This paper proposed an unsupervised domain adaptation method for 3D lidar-based object detection. The idea is simple and straightforward: using cross-domain detector + offline tracking to provide pseudo-labels, inspired by similar UDA efforts for 2D detection. Experiments are conducted over multiple self-driving perception datasets, and results validated the effectiveness of the proposed method. 

Pros:
* The idea is simple and straightforward. The approach is technically sound. 
* The presentation is clear. The introduction is easy to follow and enjoyable to read; Related work is thorough properly reflects the current states. Technical details are clearly described so that reproducing should not be very difficult. 
* The experiment showcases solid performance improvement over baselines (self-training and statistical normalization)
* The paper also conducted very detailed and convincing ablation studies. 
* Consistent improvements have been seen in several datasets and two different detectors.

Cons:
* I have some concerns regarding claimed contributions/novelty. 
* Offline tracking is not adequately benchmarked to justify the choice of extrapolation. 
* The online tracker is not on par with the current state of the art. 
* There is not enough pseudo GT quality analysis against manually labeled ground-truth. 

The usage of "video" to produce confident pseudo labels for unsupervised domain adaptation has been stressed in the introduction. However, as the related work described, this has been explored before with a similar technical approach for 2D detection (offline tracking to produce labels); see Roy-Chowdhury et al.

It's hard to say if extrapolation is a significant contribution unless adequately benchmarked, showcasing the offline tracker has improved using this trick. Such benchmarking could be done on the KITTI tracking benchmark to compare with/without extrapolation procedure. The current ablation on UDA tells little information as improvement is not significant. 

There is no comparison against other trackers. Based on the reported numbers, the online tracker adapted from Diaz-Ruiz et al. 2019 is subpar from the current state-of-the-art Kalman-based online tracker. It's hard to justify why this one is chosen. Why not Weng et al. 2020 or Chiu et al. 2020, as mentioned in the paper? In particular, the tracker in Weng et al. 2020 is open-sourced. 

Please provide an mAP evaluation of the pseudo GT quality over some sequences with GT labels. 

Although not required, it would be great to see whether the author plan to release the code. 

--------------------------------------------------------- Post-rebuttal comments ---------------------------------------------------------------------------

I carefully read the rebuttal and other reviewer comments. The author addressed my concerns on pseudo-label quality assessment and comparison against SOTA trackers. From the experimental perspective, I am very convinced the paper did a great job now. Please incorporate these additional experiments into the paper making it more complete. 

That being said, similar to other reviewers, I am not very convinced about the author's reply on novelty/contribution. It's true it has not been applied in 3D, which is new. However, I am not convinced by the claims in rebuttal, such as "using physics-based dynamics models" (I think you are referring to kinematics-based instead of physics-based), "3D extrapolations" (which could induce potential problems due to the multi-modal future uncertainty), and "self-training" (which is not new). Thus, if the paper gets accepted, I strongly encourage the author to rewrite the introduction and properly reflect the core contributions.

Overall I am still on the positive side. But I am fine with both decisions.  
review_title: Paper1349

final_score: rating: 6: Marginally above acceptance threshold
review_content: The topic of adapting 3d object detectors to new domains is important. The paper clearly motivates the problem, clearly presents the methods and shows detailed experiments.  I really enjoyed reading the paper. 

My main concern is that the two components of the method (self-training with pseudo labels and generating more pseudo labels with an object tracker for object detection) have been developed and widely used in the computer vision domain for 2d object detectors. The main novelty of this paper lies in using the counterparts of the two components in 3d for the new 3d object detection task. The use of self-training is almost the same as all previous methods. There are a few interesting engineering parts in using 3d object tracker to expand the pseudo labels such as label extrapolation and interpolation.      

Another question I have is that when the object detector gets stronger, do we need a stronger object tracking algorithm in order to provide additional useful information. If the tracking algorithm is too weak, relative to object detection methods, the augmented pseudo labels will be too noisy to provide any help. Discussions or experiments on this point would be very helpful in understanding the application domain of the proposed method.  

Although the novelty of the method is rather small, the authors have made good efforts in supporting the work with extensive experiments. The authors have evaluated their method on five datasets (all the 2 out of 5 combinations). The results are good across all the scenarios. The paper is clearly written and the method is well motivated. 

I am not sure whether a paper with extensive experiments and relatively small technical contribution should be considered as a good paper for ICLR.  After reading other reviews and the rebuttal, I opt for acceptance.   

review_title: The paper leverages two ideas for adapting 3d object detection to new domains: 1) self-training with pseudo labels and 2) using object tracking to refine and extend the initial pseudo labels by an object detector. The technical novelty is rather limited as both components have been widely used in the computer vision field. However, the paper is nicely written with detailed experiments. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros:
- The proposed method is simple yet effective and has wide uses in real-world applications
- Solid experiments across 5 benchmarks.
- This method does not rely on the source domain data and learned trackers.


Cons:
- The object detector will detect objects accurately only when they are close to the self-driving car. The claim is not supported when there is a large domain gap (e.g., different LiDARs or significantly different scenarios). The proposed model will fail to handle this situation. 
- For the static cars, why don't use ego-motion to model the temporal relationships? It should have a better performance than EKF.
- The generation of the pseudo-labels depends heavily on the confidence scores obtained from the object detector. Confidence scores > 0.8. How is the threshold of 0.80 chosen? Would other thresholds be more effective?
- Why does the author only post results in 50-80m in Tab. 3. The accurate detection in 0-50m is more important, although the relative improvement may be less.
- The method is somewhat similar to the existing tracker-based UDA methods, thus the novelty is limited. However, the application of 3D detection and the extensive experiments are great and may benefit further research significantly.
review_title: This paper addresses the ability of 3D object detection within the context of self-driving cars to adapt to new unlabeled data by using extended kalman filter in temporal sequences.

Meta-review: This paper proposed an unsupervised domain adaptation method for 3D lidar-based object detection. Four reviewers provided detailed reviews: 3 rated “Marginally above acceptance threshold”, and 1 rated “Ok but not good enough - rejection”. The reviewers appreciated simple yet effective idea, the well motivated method, the comprehensiveness of the experiments, and well written paper. However, major concerns are also raised regarding the core technical contributions on the proposed approach. The ACs look at the paper, the review, the rebuttal, and the discussion. Given the concerns on the core technical contributions, the high competitiveness of the ICLR field, and the lack of enthusiastic endorsements from reviewers, the ACs believe this work is not ready to be accepted to ICLR yet and hence a rejection decision is recommended.

Decision: Reject