Title: LiT: Unifying LiDAR "Languages" with LiDAR Translator
Venue: NeurIPS 2024
Accepted: yes
Keywords: LiDAR Translation

Domain Unification

3D Detection
Abstract: LiDAR data exhibits significant domain gaps due to variations in sensors, vehicles, and driving environments, creating “language barriers” that limit the effective use of data across domains and the scalability of LiDAR perception models. To address these challenges, we introduce the LiDAR Translator (LiT), a framework that directly translates LiDAR data across domains, enabling both cross-domain adaptation and multi-domain joint learning. LiT integrates three key components: a scene modeling module for precise foreground and background reconstruction, a LiDAR modeling module that models LiDAR rays statistically and simulates ray-drop, and a fast, hardware-accelerated ray casting engine. LiT enables state-of-the-art zero-shot and unified domain detection across diverse LiDAR datasets, marking a step toward data-driven domain unification for autonomous driving systems. Source code and demos are available at: https://yxlao.github.io/lit.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a method to help alleviate the domain gaps among different datasets with different LiDAR sensors, which can enable zero-shot detection on a new dataset. The proposed method including Scene Modeling for foreground and background reconstruction and LiDAR Modeling with statistical and ray-drop modeling. Another contribution is that the authors also accelerate the ray casting algorithm using GPU. The authors conducted single-domain and Multi-domain unification experiments on Waymo, nuScenes, and KITTI datasets, which achieves SOTA performance compared to previous works. The authors also provide ablation studies on foreground diversity and LiDAR noise injection. In addition, the authors show the run time performance after the GPU acceleration.\n\nSTRENGTHS:\nOriginality: The foreground and background reconstruction and LiDAR Modeling and the statistical and ray-drop modeling in LiDAR Modeling make the paper differ from previous works.\nQuality: The code is provided. The performance is evaluated on multiple datasets, and achieves SOTA performance compared to previous works, and ablation studies are good. The GPU acceleration is also good.\nClarity: The images in the paper are clear and easy to understand.\nSignificance: The paper demonstrates the potential of zero-shot detection on a new dataset by 3D reconstruction from multiple different dataset and LiDAR settings and LiDAR simulation.\n\nWEAKNESSES:\nIn the title of the paper, the use of terms such as \"Language,\" \"Translator,\" and \"LiT\" appears to be capitalizing on the popularity of the trending terms \"LLM\", \"ViT\", and \"DiT\", potentially misleading readers.\nSECOND and PV-RCNN are relatively old detection models, it's better to have experiments on more recent models such as CenterPoint, and other SOTA models to further demonstrate the effect of domain unification on SOTA models and even achieve new SOTA results. This would significantly enhance the paper's persuasiveness and impact.\n\nQUESTIONS:\nIn the multi-domain unified training experiments, have you considered including comparisons with other non-reconstruction semi-supervised learning methods?  (like pseudo-labels and so on)\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the significant gap between different LiDAR datasets (related to sensors, environments, etc.), this paper proposes a solution that differs from the existing model-based adaptation approach. By employing a scene-reconstruction-data-simulation approach, it achieves consistent representation of different LiDAR datasets. This data-driven method partially resolves issues such as domain shift in autonomous-driving-related 3D  point cloud learning.\n\nSTRENGTHS:\n- Innovatively analogizing the domain gap between different LiDAR data to that between languages, this paper proposes a data-driven cross-sensor training method from a \"translation\" perspective.\n\n- The proposed method shows good performance across different datasets, especially in terms of the AP3D metric.\n\n- The paper is well-written with clear logic and comprehensive experiments.\n\nWEAKNESSES:\n- Does \"foreground\" only refer to vehicles? Do pedestrians, bicycles, and similar entities fall into this category?\n\n- Similarly, in background reconstruction, is consideration limited to rigid bodies like the ground? In autonomous driving scenarios, is there no need to consider non-rigid objects such as vegetation?\n\n- In the current version, it seems that scene variations are not significant. Does this mean it's difficult to address zero-shot scenarios? For instance, if the source data are all from residential areas, is it challenging to accurately simulate point clouds from downtown areas?\n\nQUESTIONS:\n- How does the modeling accuracy of different foreground/background components affect the results of this paper?\n\n- Since the background is static, can it be replaced by other data sources? For example, historical high-precision drone point clouds or three-dimensional maps from scene reconstruction?\n\nLIMITATIONS:\nThe analysis and discussion regarding scene reconstruction need improvement, as suggested by the previous  comments.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a unifying LiDAR Translator named LiT to achieve LiDAR domain adaptation. Differing from current model-driven approaches, LiT adopts a novel data-driven approach, embedding disparate LiDAR attributes into a common representation. LiT\nproposes a generalizable scene modeling and LiDAR statistical modeling. Besides, an efficient ray-casting engine is proposed to accelerate the above models. LiT also achieves efficient SoTA performance on several LiDAR datasets.\n\nSTRENGTHS:\nS1. LiT adopts a novel data-driven approach instead of the classical model-driven approach, embedding disparate LiDAR attributes into a common representation. This research direction provides much value for real-world applications in autonomous driving industries. \n\nS2. An effective ray-casting engine is proposed to accelerate LiT on GPUs.\n\nS3. Experiments on widely used datasets demonstrate the SOTA performance of LiT.\n\nWEAKNESSES:\nW1. This work looks like a data normalization operation, only modifying different datasets into a unified representation.\n\nW2. The authors argue that model-driven approaches will introduce considerable costs associated with customizing model structure and training data for new, specific domains. However, this work has an extra LiDAR statistical modeling, this operation also causes additional costs.\n\nW3. Table 7 shows that LiT may not avoid the problem of model-driven approaches, that is, requiring different configurations for distinct datasets.\n\nQUESTIONS:\nQ1. The authors argue that model-driven approaches need extra training for new domains, while the proposed LiT also needs extra LiDAR statistical modeling. Can the authors provide a detailed comparison to prove that data-driven approaches are significantly better than traditional model-driven ones?\n\nQ2. Since datasets will be unified into a common representation, why LiT needs different training hyperparameters for distinct domain adaptation tasks, as shown in Table 7? It seemed to contradict the original motivation of this paper, i.e., unifying different types of LiDAR sensors.\n\nLIMITATIONS:\nAlthough the proposed data-driven approach seemed to be a promising research direction, LiT lacks sufficient comparisons with the model-driven approaches. Besides, LiT seemed to show \"ununified\" for the training process of different adaptation tasks.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel framework designed to unify LiDAR data into a single target “language” and unified domain detection capabilities across diverse LiDAR datasets, marking a step toward domain unification for LiDAR-based autonomous driving systems. Experiments on dataset KITTI, Waymo, and nuScenes demonstrate the superiority of the proposed method in the task of single-source and multi-sources domain adaptation.\n\nSTRENGTHS:\n1.\tThe paper is novel in introducing LiDAR Translator (LiT) to joint training across multiple datasets. LiT enables efficient state-of-the-art zero-shot and unified domain detection capabilities across diverse LiDAR datasets. \n2.\tThe paper is well-written and easy to follow, especially the part explaining the background. \n3.\tIt presents good experimental results and intuitive visualizations, convincingly demonstrating its effectiveness.\n\nWEAKNESSES:\n1.\tThe motivation of this paper is not clear. If it is possible to accurately model the target domain data, why is there a need to translate the source domain data into the target domain data?\n2.\tAs the core component of this work, the translator requires more direct experimental validation, such as measuring the distributional differences between the translated data and the target data, rather than solely relying on verification through downstream domain adaptation tasks.\n3.\tIt  lacks of comparative experiments with the latest state-of-the-art methods.\n\nQUESTIONS:\n1.\tHow can we ensure the accuracy of modeling the target data? Will the differences between simulated data and real data have negative impacts?\n2.\tModeling the target LiDAR data and target scenes requires a lot of prior information. When this information is unknown, how can we use the method proposed in this paper to solve the domain adaptation problem? From my understanding, the objective of domain adaptation is to address the challenge of having limited data or labels in the target domain.\n\nLIMITATIONS:\nThe authors discussed potential limitations about the data, annotation and the category of object.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}