LiT: Unifying LiDAR "Languages" with LiDAR Translator

Authors: yixing  lao
tao  tang
xiaoyang  wu
peng  chen
kaicheng  yu
hengshuang  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 16:11:32

Abstract: LiDAR data exhibits significant domain gaps due to variations in sensors, vehicles, and driving environments, creating “language barriers” that limit the effective use of data across domains and the scalability of LiDAR perception models. To address these challenges, we introduce the LiDAR Translator (LiT), a framework that directly translates LiDAR data across domains, enabling both cross-domain adaptation and multi-domain joint learning. LiT integrates three key components: a scene modeling module for precise foreground and background reconstruction, a LiDAR modeling module that models LiDAR rays statistically and simulates ray-drop, and a fast, hardware-accelerated ray casting engine. LiT enables state-of-the-art zero-shot and unified domain detection across diverse LiDAR datasets, marking a step toward data-driven domain unification for autonomous driving systems. Source code and demos are available at: https://yxlao.github.io/lit.

Main paper: 
 introduction: 
 LiDAR provides accurate geometry measurements of the surrounding environment, making it one of the most popular sensor choices for various autonomous systems [1, 2, 3, 4, 5] . Despite its importance, perception models trained on a specific LiDAR setup often struggle to generalize across different sensors [6, 7, 8] , primarily due to the domain gaps attributable to variations in sensor intrinsic characteristics and driving environments. These gaps, akin to "language barriers," significantly degrade model performance when transferring between different LiDAR sensors and hinder effective joint training across multiple datasets. In a real-world scenario, it takes tremendous time and resources to collect, annotate, and train just one model for a specific LiDAR setup [9, 10, 11] . Consequently, this barrier acts as a bottleneck in training scale-up, which has been a driving force behind recent rapid advancements in other areas of representation learning, such as 2D vision [12, 13, 14, 15] and natural language processing (NLP) [16, 17, 18, 19] . Addressing the "language barriers" in the context of LiDAR data is therefore crucial for unlocking similar advancements within autonomous driving.Recent efforts [6, 20] have already acknowledged the significance of "language barriers" within LiDAR data as a meaningful challenge. However, solutions to date primarily focus on model-driven approaches, attempting to bridge these gaps through adaptations within the perception model itself. While these methods have shown promise in bridging domain gaps in certain contexts, they introduce a notable drawback: the cost of customizing model structure and training data for new, specific domains. This requirement demands significant resources and limits the scalability and flexibility of deploying autonomous systems across varied environments.To this end, we present the LiDAR Translator (LiT), a novel data-driven approach that functions similarly to a language translator, converting disparate LiDAR "dialects" into a common "language" (Fig. 1 ), and further unleashing the zero-shot capacity and joint training potential of LiDAR-based models. The overall pipeline is available in Fig. 4 . Specifically, LiT reconstructs the source domain scene with neural implicit representations, capturing both the static background and dynamic foreground elements (Sec. 4.1). Then, LiT simulates the target domain LiDAR sensor model using a custom GPU-accelerated ray casting engine, generating translated LiDAR scans that faithfully replicate the target domain's characteristics (Sec. 4.2) . By doing so, LiT effectively bridges the gap between different LiDAR "languages," enabling seamless domain adaptation and unification of disparate LiDAR datasets. Through this well-designed framework, LiT effectively bridges the gap between different LiDAR "languages," facilitating seamless domain unification of LiDAR datasets.LiT advances domain adaptation through data-side unification and multi-domain joint learning [6] . We show in our experiments that, in single-source adaptation scenarios, LiT demonstrates superior zero-shot detection performance compared to existing model-driven and data-driven techniques. Additionally, by translating multiple data sources into a unified target domain, LiT facilitates multi-dataset joint training, where combining data from various sources not only improves model performance beyond single-source training but also establishes a new paradigm for data-centric learning strategies.2 Related work 3D representation learning. In the realm of autonomous driving, achieving the unification of perception models for LiDAR data across different domains presents a distinct challenge, one that diverges from the significant progress observed in 2D domains where large-scale pre-training has dramatically enhanced downstream task performance [21] . Specifically in the domain of 3D representation learning, the field is still in an exploratory phase. Many existing studies have focused on training models from scratch, tailored to specific datasets [22] . Initially, this research largely targeted the recognition of individual objects [23, 24, 25, 26, 27] . Over time, it has evolved to address the more complex task of interpreting real-world, scene-centric point clouds [22, 28, 29, 30] , marking a notable advance in our understanding of 3D data. Inspired by the achievements of large-scale representation learning and the effectiveness of multi-dataset synergistic approaches, such as Point Prompt Training (PPT) [6] , LiT aims to overcome the constraints of model-side adaptations by utilizing a data-driven approach to bridging domain gaps.LiDAR domain adaptation. LiDAR domain adaptation is critical for enabling models trained on one domain to effectively operate in another, accommodating variations such as sensor types and environmental conditions [31, 32, 33, 34, 35] . Techniques have varied widely: SN [36] tackles object size discrepancies through normalization based on object statistics; ST3D [7] and ST3D++ [8] utilize a self-training pipeline with pseudo-labels for fine-tuning; LiDAR-Distillation [37] mitigates beam-induced domain shifts with a progressive framework; SPG [38] aims to fill in missing points in foreground areas; and 3D-CoCo [39] employs domain-specific encoders and contrastive learning for transferable representation. LiT aligns with these efforts by employing zero-shot object detection as a primary benchmark to measure its effectiveness in domain adaptation. However, LiT's ambition extends beyond mere adaptation; it aims to unify the "languages" of multiple LiDAR domains, exploring the synergistic potential of such unified data.Autonomous driving simulator. Simulators play an important role in autonomous driving research by making data collection and annotation much easier and cheaper. Initial efforts leveraged ground-truth information extracted from graphic engines, such as CARLA [40] , by utilizing various simulators [41, 42, 43, 44, 45] . Despite their utility, these sim-to-real approaches often involve complex dependencies on simulator engines, which can be cumbersome and costly to establish, and still result in a notable sim-to-real gap. In response to these challenges, subsequent studies like LiDARsim [46] and PCGen [47] have attempted to bridge this gap by simulating point clouds from real data through multi-step, data-driven pipelines. More recent advances, including UniSim [48] and LiDAR-NeRF [49] , have explored the use of neural implicit fields to simulate new sensor views. However, these methods primarily generate data reflective of the source domain and do not adequately address variations across datasets. Different from these works, LiT enables direct translation of real-world source data into target domain LiDAR characteristics, effectively bridging the domain gap through simulation.3 Pilot study 
exploring language barriers among lidars: 
 The progression of autonomous driving technologies is intimately tied to the diversity and quality of LiDAR data, which provides a detailed three-dimensional representation of environments. Operating in varied settings, autonomous vehicles utilize diverse LiDAR sensors, each characterized by unique specifications like field of view, beam count, and ray-drop behavior. This results in notable variations across datasets such as Waymo [10] , nuScenes [11] , and KITTI [9] , creating significant domain gaps akin to language barriers between LiDAR "languages".These gaps manifest in various forms, including alterations in point cloud density due to beam count differences and changes in spatial coverage from FoV variations. Additionally, scene content differences (such as vehicle sizes, orientations, and types) mirror the geographical and operational diversity of each dataset, adding layers of complexity to the domain-specific challenges (as visualized in Fig. 2 ). Such disparities obstruct the direct application of models trained on one dataset to another (see Table 1 ) and further pose significant obstacles to exploring synergistic training across these datasets. By tackling these barriers head-on, LiT aims to unlock the potential of data scale.We wish to break the language barriers among LiDARs with our LiDAR Translator.Previous attempts to bridge domain gaps [7, 8] result, while they may narrow the domain gap to a degree, they do not fully exploit the potential of diverse datasets to bolster model robustness and generalization.Contrary to model-side adaptation, data-driven adaptation presents a more practical and efficient avenue for LiDAR domain unification without the need for additional adjustment on the base model. By translating LiDAR data from various source domains into a standardized target domain "language", we can unify discrepancies across datasets. This harmonization creates a cohesive dataset that reflects the diversity of multiple sources, addressing the issue of negative transfer and enriching the training data to boost model performance and generalizability in different settings.Specifically, if we can translate data from an available source domain to an unseen target domain, we can further use the translated data for model training and fulfill downstream tasks on the target domain, effectively achieving zero-shot cross-domain learning. Moreover, if we can unify multiple source domains into one target domain, model training can further take advantage of a larger scale of data training to push the model's generalization capabilities to a higher level, as illustrated in Fig. 3 .In a word, embracing data-driven domain unification has great potential in both zero-shot adaptations from source to target and large-scale multi-domain joint training.We treat data-driven domain unification as the original point of our LiDAR Translator. 2) with GPU-accelerated ray casting. LiT is highly efficient, as it can translate a multi-frame LiDAR scene in typically less than one minute (Table 6 ).4 LiDAR translator 4.1 Scene modeling Foreground modeling. Previous work ReSimAD [51] relies on hand-crafted 3D vehicle assets from simulator engines [40] , which limits the diversity of the foreground objects and the realism of the simulated data. In contrast, LiT reconstructs the foreground objects from multi-frame LiDAR point clouds, which are more representative of real-world data with diverse vehicle geometries. To achieve this, we first track the vehicles across multiple LiDAR frames, where the tracking information is typically provided by LiDAR datasets [9, 10, 11] . Inspired by the recent success of implicit neural representations [52, 53] in reconstructing shapes from LiDAR point clouds, we train an SDF-based reconstruction model on the ShapeNetV2 [54] vehicle category and apply it to reconstruct the foreground objects from multi-frame LiDAR point clouds. With this approach, we can accurately reconstruct the detailed geometry of diverse vehicle shapes. As shown in our experiments in Table 5 , the increase in foreground diversity leads to a significant performance increase in zero-shot detection tasks. We refer the reader to Sec. A.1 for details and visualizations on the foreground modeling.Background modeling. The primary goal of background modeling is to faithfully reconstruct the background environment. However, the efficiency and scalability of the background modeling method are equally important, especially when dealing with joint training on multiple large-scale datasets.ReSimAD [51] uses NeuS [55] for background reconstruction, which requires custom training for each scene with time measured in hours. In contrast, LiT models the scene as the zero-level set of a 3D implicit field defined by a hierarchical neural kernel field [56, 57] . This approach is generalizable, eliminating the need for retraining on every new LiDAR sequence and significantly reducing LiT's full-scene translation time to less than a minute for a 200-frame LiDAR scene, as detailed in Table 6 . This efficiency plays a crucial role in enabling the scalability of LiT across various datasets. We refer readers to Sec. A.2 in the supplementary material for further details and visualizations on background modeling.Module adaptability. Both the foreground and background modeling are crucial modules for achieving high fidelity in scene representation, enabling accurate domain adaptation. In LiT, these modules are designed to be swappable, allowing for the integration of more advanced scene representation methods as they are developed. While the specific modeling techniques are important, our primary contribution lies in how these components work together within our domain translation framework to enable effective domain unification. 
lidar modeling: 
 Modeling the different LiDAR characteristics is a critical step in domain adaptation, as it directly affects the simulated data distribution and characteristics of the target domain. Unlike existing work [51] that relies on full-scale simulators, we develop a custom LiDAR modeling and ray casting pipeline that is more flexible and efficient, and it does not require complex dependencies when using simulator engines. We model the LiDAR characteristics of beam angles. All of these are made possible by our hardware-accelerated LiDAR ray casting engine. The naive implementation for LiDAR ray generation is to evenly distribute the rays in the horizontal axis, based on the LiDAR's horizontal field of view (FOV) and beam counts. However, this approach does not accurately reflect the real-world LiDAR characteristics, which are often non-homogeneous in the vertical direction, as shown in the top row of Fig. 2 . To address this issue, we conduct a statistical analysis of LiDAR rays, focusing on the ray distribution in the vertical axis. First, for each LiDAR ray in the dataset, we compute its vertical angle based on the ray's xy-plane distance and z-axis distance. Then, we calculate the distribution of the vertical angles and identify the peak angles where the majority of the LiDAR rays are concentrated. Specifically, this is done by finding H statistical modes in the distribution of vertical angles, where H is the height of the range image or the number of beams in the LiDAR sensor. Notably, we do not require manual labeling of target domain data, only a small amount of target domain LiDAR data to perform this statistical analysis. After the modeling, our LiDAR ray generation module then generates rays based on these identified peak angles, ensuring that our simulated LiDAR data is representative of real-world LiDAR characteristics. Fig. 5 shows the average Chamfer distance between the real-world LiDAR data and the self-to-self translated LiDAR data on nuScenes. The chamfer distance is reduced when statistical modeling is applied (blue curve) compared to the vanilla case (red curve), indicating that the simulated LiDAR data more closely resemble the real-world LiDAR data.LiDAR ray-drop modeling. Ray-drop of LiDAR rays is frequent in real-world scenarios due to various factors, including object reflectivity, absorption, or occlusion. We model this phenomenon by training a ray-drop predictor model that predicts the likelihood of a ray being dropped. To collect the ray-drop dataset, we reconstruct background and foreground scenes and perform ray casting using the dataset's own LiDAR parameters. For each ray, we record whether it hits a mesh or not in the simulated scene. Inspired by PCGen [47] , we collect the ray direction r, the point distance d, and the ray incident angle θ for each ray. Then, we train an MLP-based ray-drop model with positional encoding [58] , to predict the likelihood of a ray being dropped. The MLP network takes (r, d, θ) with positional encoding as input, and outputs the probability of the ray being dropped.For quantitative evaluation, we perform a self-to-self translation task on the nuScenes dataset to assess the quality of the ray-drop modeling. Fig. 5 shows the chamfer distance decreases when ray-drop modeling is applied, and it shows the combined effects of applying both statistical modeling and ray-drop modeling together. For qualitative evaluation, we present the visualization of the ray-drop model trained on the nuScenes dataset in Fig. 6 . The simulated LiDAR points, especially those proximal to vehicles, effectively resemble the nuScenes dataset's characteristic patterns. Hardware accelerated LiDAR ray casting. While existing work [51] relies on integrating with full-scale simulators like CARLA [40] , which can be cumbersome to recompile and less efficient to run, we have developed a hardware-accelerated LiDAR ray casting engine for LiT that offers greater flexibility and efficiency. Specifically, our LiDAR ray casting engine emits rays based on the LiDAR modeling, computes the location of the ray-mesh hit point, the surface normal at that point, and the incident angle of the ray, providing the necessary data to simulate target domain points. Given the surface normal n of the triangle hit by the ray and the ray direction r, the incident angle θ is computed as θ = arccos ((r • n)/(∥r∥∥n∥)). LiT's ray casting engine is accelerated on both CPU with Intel Embree and GPU with Nvidia OptiX, enabling real-time ray casting (22Hz to 31Hz) when running on a single GPU. The efficiency of this engine allows LiT to translate a multi-frame LiDAR scene typically in less than one minute, making large-scale domain unification tasks feasible. For more detailed runtime statistics, refer to Table 6 .Flexible composition of LiDARs and scenes. LiT's flexible pipeline allows seamless deployment of LiT-modeled LiDARs to new environments that are not necessarily modeled by LiT. As demonstrated in Fig. 7 , we can deploy a LiT-modeled LiDAR in nuScenes's style in a given new environment, such as the Mai City [59] scene. This flexibility allows LiT to work with various sources of reconstructed 3D maps, while being able to simulate LiDAR scans with target LiDAR sensor characteristics, which is particularly useful for deploying new LiDARs or simulating out-of-distribution scenarios. 
experimental settings: 
 Datasets. Our experiments are conducted on three widely utilized datasets for autonomous driving research: KITTI [9] , Waymo [10] , and nuScenes [11] . These datasets present a diverse range of LiDAR characteristics, including varying beam counts and sensor configurations, making them ideal for evaluating our LiDAR translation approach, LiT. Adaptation scenarios covered include cross-beam-count adaptations such as Waymo → nuScenes (high-beam to low-beam) and nuScenes → KITTI (low-beam to high-beam), as well as adaptations within the same beam count but different LiDAR characteristics like Waymo → KITTI. Table 6 provides an overview of the dataset statistics, including details on LiDAR beams and the distribution of foreground objects.Evaluation metrics. To evaluate how well our translated point clouds match the statistical distribution of the target domain, we follow LiDARGen [60] and UltraLiDAR [61] to compute Maximum-Mean Discrepancy (MMD) and Jensen-Shannon divergence (JSD) in BEV voxel occupancy between the source domain point clouds translated to the target domain style and the actual target domain point clouds. As the source and target domains come from entirely different scenes, we measure the overall distributional similarity rather than per-scene alignment, providing a quantitative measure of how well LiT captures the general characteristics of the target domain.The downstream detection task evaluation follows the KITTI benchmark standards, focusing on the car category. We report Average Precision (AP) across 40 recall positions at an Intersection over Union (IoU) threshold of 0.7 for both bird's eye view (BEV) and 3D detection. For KITTI, metrics are reported for front-view annotations only, while for Waymo and nuScenes, we extend our evaluation to include full ring-view point clouds generated from top-mounted LiDAR sensors, while other LiDARs are not used in our experiment. The evaluations are conducted using the SECOND-IoU [50] and PV-RCNN [62] models, following the evaluation protocol of previous works [36, 7, 51] .Baselines and comparison. For the domain unification tasks, LiT is evaluated against key baselines in autonomous driving domain adaptation: (i) SN [36] , which normalizes object sizes using target domain statistics; (ii) ST3D [7] , employing iterative pseudo label generation and curriculum-based training; (iii) ReSimAD [51] , a method combining reconstruction and simulation; and (iv) Oracle, the theoretical performance upper bound via full target domain supervision.Statistical alignment with target domain. To validate the effectiveness of our LiDAR translation approach directly, we compare the distributional differences between translated and ground-truth target domains, as shown in Table 2 . Across all adaptation scenarios, LiT significantly reduces both MMD and JSD metrics compared to the baseline, which is the LiDAR data from the source domain. More substantial improvements are observed in cross-beam-count scenarios (64-beam → 32-beam and 32-beam → 64-beam), where LiT achieves 3 ∼ 4× reduction in MMD BEV .Single source domain unification. In the single source domain adaptation setting as shown in Table 3 , LiT consistently outperforms all baselines, including the model-based adaptation approach ST3D and data-based adaptation approach ReSimAD, in both the AP BEV and AP 3D metrics across different settings. Notably, in the challenging nuScenes → KITTI tasks, where the low-beam LiDAR Table 4 : Multi source domain unification. We compare models trained with one, two, and three source domains. In the table, W = Waymo, N = nuScenes, and K = KITTI. We report the AP BEV and AP 3D of the car category at IoU = 0.7. The last row represents the "oracle" K → K model trained with full target-domain supervision. Remarkably, in the W + N → K task, LiT surpasses the oracle performance in SECOND-IoU's AP BEV even when the target domain is never seen during training. When all three source domains are used, LiT surpasses the oracle performance in all metrics. Foreground diversity. The diversity of foreground objects plays an important role in the domain adaptation's performance. Specifically, when transitioning from a single shared foreground mesh to 50 shared meshes sees an improvement in both AP BEV and AP 3D . Furthermore, with only foreground simulation, LiT still manages to achieve reasonable performance, showcasing LiT's robustness. The results are shown in Table 5 . Scene modeling inaccuracies. To understand how modeling accuracy affects performance, we simulate inaccuracies by adding Gaussian noise with std = 0.01m, 0.02m, and 0.05m to the reconstructed mesh vertices of foreground-only, background-only, and both foreground and background. As shown in Table 5 , the detection model is more sensitive to inaccuracies in the foreground compared to background, which is expected as foreground objects are the main target for object detection. Even with scene modeling inaccuracies, LiT still substantially outperforms the source-only (no translation) baseline, demonstrating the robustness of our approach.Runtime performance. Table 6 shows the runtime performance and key statistics of LiT's LiDAR translation process. We report the average runtime and data statistics of 50 scenes from each dataset. The runtime is measured on a desktop PC with a single NVIDIA RTX 4090 GPU. Notably, LiT is capable of translating a full multi-frame LiDAR scene from one domain to another in under a minute.After the reconstruction steps are done, the LiDAR ray casting process is highly efficient and can run in real-time. The efficiency of LiT demonstrates its scalability in real-world applications, where the ability to quickly adapt sensor data to different domains can enhance perception systems. 
conclusion: 
 In conclusion, we present the LiDAR Translator (LiT), a pioneering framework that unifies LiDAR data into a common "language" in a comprehensive system consisting of scene modeling, LiDAR modeling, and domain adaptation. LiT overcomes domain discrepancies in LiDAR data from diverse sensor setups and environments, enabling seamless integration of multiple LiDAR datasets, enhancing zero-shot detection capabilities, and improving the representation quality of pre-trained models across a variety of scenarios. Additionally, LiT's efficiency streamlines data preprocessing, reducing both time and computational demands, and facilitating quicker development cycles in autonomous systems. This acceleration promotes rapid advancements and the deployment of safer, more effective vehicles.Social impact and limitations. The LiT framework significantly advances autonomous driving by facilitating domain adaptation for LiDAR data, which could enhance transportation safety and efficiency. However, it has limitations, such as its exclusive reliance on LiDAR data, its dependency on annotated source datasets for foreground reconstruction, and its current focus on vehicle objects solely. Future enhancements should aim to incorporate other object types and sensor modalities, as well as reduce dependency on extensive labeling, to broaden LiT's applicability and impact. A.1 Foreground modeling detailsTracking and fusion. Reconstructing foreground objects directly from each LiDAR frame is suboptimal as it does not take into account the multiple viewing angles of the object as the LiDAR sensor or the object moves. To address this issue, we fuse the point clouds from multiple LiDAR frames to reconstruct the foreground objects. Typically, tracking information is provided as unique object identifiers in the LiDAR datasets [9, 10, 11] , where vehicles are tracked across multiple frames using these identifiers. For each identified vehicle, its point clouds from different frames are aligned based on their bounding box information and then fused.Mesh reconstruction. With multiple LiDAR frames, the fused point clouds are incomplete and noisy due to the sparsity and scanning gaps inherent in LiDAR data. To faithfully reconstruct the foreground objects while addressing the noise and sparsity issues, we need to introduce prior knowledge of vehicle geometries. Inspired by the success of neural implicit surface reconstruction for object shape modeling [63, 52] , we first train an SDF-based model with the ShapeNetV2 [54] vehicle category and then apply this model to reconstruct the foreground object meshes. In essence, given a latent code, the model maps 3D points to a signed distance field (SDF) representing the mesh surface, where the key here is to solve for the latent code given multi-view point cloud observations.Given a set of points from LiDAR scans, {x i } N i=1 , our goal is to find a latent code, z, that best represents the underlying surface that these points belong to. This optimization process is guided by minimizing the Signed Distance Function (SDF) loss, defined as:L SDF = 1 N N i=1 (∥f θ (z, x i ) -0∥ 1 ) + λ∥z∥ 2 2 ,where f θ (z, x i ) denotes the predicted SDF value for point x i given the latent code z, and the target SDF value for points on the object surface is set to zero. The term λ∥z∥ 2 2 adds an L2 regularization on the latent code to encourage generalization by penalizing its magnitude, with λ being the regularization coefficient.Notably, as our mesh is reconstructed from point clouds, it is not limited to using human-created vehicle 3D assets, which is a key difference from ReSimAD [51] . This optimization process effectively adapts the generic vehicle geometry learned from ShapeNet to fit the specific geometry of the vehicle represented in the sparse and noisy LiDAR point cloud. By iteratively refining the latent code, we can generate a mesh that accurately captures the detailed geometry of the vehicle, even in the presence of data sparsity and noise inherent to LiDAR scans. We show some example visualizations of foreground modeling in Fig. 8 .The hyperparameters for scene modeling are shown in Table 7 . For background modeling, while most parameters are consistent across both Waymo and nuScenes datasets, a key difference is their frame rates. Specifically, every other frame is skipped in the Waymo dataset, which is attributed to its denser point clouds and a higher LiDAR frame rate of 10Hz, in contrast to the 2Hz frame rate of nuScenes. Consequently, we sample the Waymo dataset at 5Hz to ensure a comparable point cloud density and maintain reconstruction quality across datasets.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper does not pose such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite and respect the licenses of all assets used in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Next, we reconstruct the background scene mesh with point clouds from multiple LiDAR frames. For each LiDAR frame point cloud, foreground objects are removed based on the bounding box information. Then, point clouds from multiple frames are transformed and fused in world coordinates by the ego vehicle's pose and LiDAR-to-ego pose.To reconstruct the background scene mesh from the fused point cloud, we follow neural kernel field [57] , where the reconstructed shape is modeled as the zero level set of a 3D implicit field f θ : R 3 → R defined as a hierarchical neural kernel field. This field combines positive definite kernels, weighted and conditioned on the inputs, and centered at the midpoints of voxels within the predicted hierarchy:f θ (x|X in , N in ) = i,l α (l) i K (l) θ (x, x (l) i |X in , N in ),where α (l) i are scalar coefficients at the i th voxel at level l in the hierarchy. Then, the kernel for the l th level, K (l) θ , is defined as:K (l) θ (x, x ′ ) = ⟨ϕ (l) θ (x|X in , N in ), ϕ (l) θ (x ′ |X in , N in )⟩ • K (l) b (x, x ′ ).Given the voxel hierarchy and predicted normals, the implicit surface is computed by minimizing a specific loss function to find optimal coefficients. This process aims to align the neural kernel field gradient with the normals at voxel centers and to approximate zero at all input points. We employ the pre-trained kitchen-and-sink model which is able to generalize to diverse datasets. For a fair comparison, the model has not been trained on the LiDAR datasets that we use in this work. Compared with ReSimAD with their NeuS [55] -based background modeling, our approach is more efficient as it does not require custom training for each scene.The hyperparameters for scene modeling are shown in Table 7 . Additional visualizations of the background modeling are provided in Fig. 9 . For foreground modeling, we utilize a fixed number of steps and a predefined learning rate to reconstruct the foreground objects. We use the same parameters for both Waymo and nuScenes datasets.As discussed in Sec. 4.2, we conduct a statistical modeling of LiDAR rays to model the distribution of the vertical angles. We identify the peak angles where the majority of the LiDAR rays are concentrated and generate rays based on these identified peak angles. This ensures that our simulated LiDAR data is representative of real-world LiDAR characteristics. We illustrate this effect visually in Fig. 10 , while the LiDAR statistics of each dataset are presented in Fig. 2 .We perform evaluation on the quality of the scene and LiDAR modeling as an integrated system. To do this, we perform a self-to-self translation task on the source domain, where we use LiT to reconstruct the scene and perform LiDAR ray casting using the same dataset's LiDAR parameters. We then calculate the Chamfer distance between the original point cloud P gt and the ray casted point cloud P sim :CD(P sim , P gt ) = 1 |P sim | x∈Psim min y∈Pgt ∥x -y∥ 2 2 + 1 |P gt | y∈Pgt min x∈Psim ∥y -x∥ 2 2 .The distributions of the Chamfer distances for the Waymo and nuScenes datasets are shown in Fig. 11 .Waymo nuScenes KITTI Raycat at 20 MetersWe provide hyperparameters used for training in Table 8 . Justification: We provide detailed information on the dataset, hyperparameters, and evaluation metrics in the main paper and supplemental material. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the societal impacts of our work in the conclusion section of the paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: All reviewers recommended acceptance, with two borderline accept, one weak accept, and one accept, This AC sees no reason to override the collective recommendations of the reviewers.

Reviews: review: paper_summary: paper_summary: The paper presents a novel framework designed to unify LiDAR data into a single target “language” and unified domain detection capabilities across diverse LiDAR datasets, marking a step toward domain unification for LiDAR-based autonomous driving systems. Experiments on dataset KITTI, Waymo, and nuScenes demonstrate the superiority of the proposed method in the task of single-source and multi-sources domain adaptation.
strength_weakness: strength_weakness: 1.	The motivation of this paper is not clear. If it is possible to accurately model the target domain data, why is there a need to translate the source domain data into the target domain data?
2.	As the core component of this work, the translator requires more direct experimental validation, such as measuring the distributional differences between the translated data and the target data, rather than solely relying on verification through downstream domain adaptation tasks.
3.	It  lacks of comparative experiments with the latest state-of-the-art methods.
questions: questions: 1.	How can we ensure the accuracy of modeling the target data? Will the differences between simulated data and real data have negative impacts?
2.	Modeling the target LiDAR data and target scenes requires a lot of prior information. When this information is unknown, how can we use the method proposed in this paper to solve the domain adaptation problem? From my understanding, the objective of domain adaptation is to address the challenge of having limited data or labels in the target domain.
limitations: limitations: The authors discussed potential limitations about the data, annotation and the category of object.
paper_summary: paper_summary: This paper proposed a unifying LiDAR Translator named LiT to achieve LiDAR domain adaptation. Differing from current model-driven approaches, LiT adopts a novel data-driven approach, embedding disparate LiDAR attributes into a common representation. LiT
proposes a generalizable scene modeling and LiDAR statistical modeling. Besides, an efficient ray-casting engine is proposed to accelerate the above models. LiT also achieves efficient SoTA performance on several LiDAR datasets.
strength_weakness: strength_weakness: W1. This work looks like a data normalization operation, only modifying different datasets into a unified representation.

W2. The authors argue that model-driven approaches will introduce considerable costs associated with customizing model structure and training data for new, specific domains. However, this work has an extra LiDAR statistical modeling, this operation also causes additional costs.

W3. Table 7 shows that LiT may not avoid the problem of model-driven approaches, that is, requiring different configurations for distinct datasets.
questions: questions: Q1. The authors argue that model-driven approaches need extra training for new domains, while the proposed LiT also needs extra LiDAR statistical modeling. Can the authors provide a detailed comparison to prove that data-driven approaches are significantly better than traditional model-driven ones?

Q2. Since datasets will be unified into a common representation, why LiT needs different training hyperparameters for distinct domain adaptation tasks, as shown in Table 7? It seemed to contradict the original motivation of this paper, i.e., unifying different types of LiDAR sensors.
limitations: limitations: Although the proposed data-driven approach seemed to be a promising research direction, LiT lacks sufficient comparisons with the model-driven approaches. Besides, LiT seemed to show "ununified" for the training process of different adaptation tasks.
paper_summary: paper_summary: To address the significant gap between different LiDAR datasets (related to sensors, environments, etc.), this paper proposes a solution that differs from the existing model-based adaptation approach. By employing a scene-reconstruction-data-simulation approach, it achieves consistent representation of different LiDAR datasets. This data-driven method partially resolves issues such as domain shift in autonomous-driving-related 3D  point cloud learning.
strength_weakness: strength_weakness: - Does "foreground" only refer to vehicles? Do pedestrians, bicycles, and similar entities fall into this category?

- Similarly, in background reconstruction, is consideration limited to rigid bodies like the ground? In autonomous driving scenarios, is there no need to consider non-rigid objects such as vegetation?

- In the current version, it seems that scene variations are not significant. Does this mean it's difficult to address zero-shot scenarios? For instance, if the source data are all from residential areas, is it challenging to accurately simulate point clouds from downtown areas?
questions: questions: - How does the modeling accuracy of different foreground/background components affect the results of this paper?

- Since the background is static, can it be replaced by other data sources? For example, historical high-precision drone point clouds or three-dimensional maps from scene reconstruction?
limitations: limitations: The analysis and discussion regarding scene reconstruction need improvement, as suggested by the previous  comments.
paper_summary: paper_summary: In this paper, the authors propose a method to help alleviate the domain gaps among different datasets with different LiDAR sensors, which can enable zero-shot detection on a new dataset. The proposed method including Scene Modeling for foreground and background reconstruction and LiDAR Modeling with statistical and ray-drop modeling. Another contribution is that the authors also accelerate the ray casting algorithm using GPU. The authors conducted single-domain and Multi-domain unification experiments on Waymo, nuScenes, and KITTI datasets, which achieves SOTA performance compared to previous works. The authors also provide ablation studies on foreground diversity and LiDAR noise injection. In addition, the authors show the run time performance after the GPU acceleration.
strength_weakness: strength_weakness: In the title of the paper, the use of terms such as "Language," "Translator," and "LiT" appears to be capitalizing on the popularity of the trending terms "LLM", "ViT", and "DiT", potentially misleading readers.
SECOND and PV-RCNN are relatively old detection models, it's better to have experiments on more recent models such as CenterPoint, and other SOTA models to further demonstrate the effect of domain unification on SOTA models and even achieve new SOTA results. This would significantly enhance the paper's persuasiveness and impact.
questions: questions: In the multi-domain unified training experiments, have you considered including comparisons with other non-reconstruction semi-supervised learning methods?  (like pseudo-labels and so on)
limitations: limitations: Yes.

Comments: title: Gentle Reminder: Review of Rebuttal & Final Score
Summarized Responses and Gentle Reminder
Summarized Responses and Gentle Reminder
Updated Response to Q1
Updates on W2. Inclusion of Newer Detection Models
comment: Dear Reviewer,

Thank you for your thorough review and insightful advice, including your suggestion regarding the naming scheme. We are pleased that you find our work valuable for the community. Thank you!

Sincerely,

Authors
Dear Reviewer,

Thank you for your thoughtful feedback and for updating your scores. We believe that your suggestions, along with our newly added experiments, have greatly enhanced the persuasiveness and completeness of our work.

Sincerely,

Authors
Dear Reviewer,

We appreciate the insights and feedback you have provided on our submission, LiT, especially regarding the inclusion of a direct evaluation of translation quality (W2). We have added MMD and JSD metrics to directly evaluate the translation quality, and we believe this has significantly improved the robustness of our work.

Similarly, we have addressed all other concerns raised in your review through detailed responses and additional experiments, and we are eager to **hear your thoughts** on whether these have adequately addressed the issues you highlighted, as well as **assigning a final score**. We are happy to discuss any further questions or provide additional information if needed. Your feedback is invaluable to us, and we look forward to your response.

Thank you again for your time and consideration.

Sincerely,

Authors
Thank you for your rebuttal and kindly summarization of the responses. I have already raised my scores.
Thank you for the detailed rebuttal and updates. Among the names you provided, I prefer "LidarUnifier" or "LidarAdapter," as they capture the essence of your method well and won't cause confusion with other methods. For example, "LidarTransformer" could be confused with [1] in the normal detection/segmentation area instead of the domain adaptation area. The additional experiments with CenterPoint strengthen the paper, demonstrating the effectiveness of your approach across newer models. Maintain the score.

References

[1] Z. Zhou et al., "LiDARFormer: A Unified Transformer-based Multi-task Network for LiDAR Perception," 2024 IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan, 2024, pp. 14740-14747, doi: 10.1109/ICRA57147.2024.10610374.
Dear Reviewer Q1U8,

We sincerely thank you for acknowledging the strengths of LiT's data-driven approach to bridging the domain gap. We are also pleased to hear that the new experiments we introduced have addressed the concerns you highlighted. This reaffirms the effectiveness of our approach and its applicability across different LiDAR datasets. 

We would like to especially _thank you_ for your suggestion to include an analysis of the inaccuracies in foreground and background modeling, respectively. This has improved our insight into how noise in the translated foreground and background can affect the model's performance. This insight is very valuable, and we are grateful for your suggestion.

Sincerely,

Authors
Dear Reviewer Kzeh,

Thank you for your detailed review and insightful questions regarding our work. Especially, we would like to take this opportunity to _thank you_ for the suggestion to include an analysis of the _cost of LiDAR modeling (W2)_. With the new experiments added, we believe this suggestion is genuinely valuable for improving the compellingness of our work.

Below, we would like to offer a **summarized version** of our responses to ensure clarity:

- **W1. About Dataset Normalization**
  - "Normalizing" datasets into a unified representation is non-trivial due to inherent differences in LiDAR sensor patterns. LiT effectively addresses these challenges by modeling the target domain LiDAR pattern with minimal unannotated data, showing superior performance over state-of-the-art domain adaptation methods like ReSimAD. We show that LiT enables efficient zero-shot detection across diverse datasets (Table 2 in the main paper), and LiT is able to combine data from multiple source domains to achieve better performance on the target domain (Table 3 in the main paper). This data-driven approach is orthogonal to traditional model-based methods and provides a practical solution for real-world applications.
- **W2. Cost of LiDAR Modeling is Minimal**
  - We add new experiments to show that LiDAR modeling can be achieved with minimal unannotated target domain data with fast run times, demonstrating cost-efficiency and practicality for real-world applications. The detailed table is provided in the original rebuttal.
- **W3. Hyperparameter Consistency**
  - We clarify that in Table 7, for a given model, the hyperparameters are actually _consistent_ across all datasets. The hyperparameters are only different when a different model is used.
- **Q1. Data-driven vs Model-driven Approaches**
  - LiT surpasses traditional model-driven approaches (e.g., ST3D) across all tested translation scenarios.
  - Besides, we see LiT's data-driven approach as a _complementary and orthogonal_ strategy to model-driven methods. LiT offering a new path forward in domain adaptation research.
- **Q2. Hyperparameter Consistency**
  - We clarify that in Table 7, for a given model, the hyperparameters are actually _consistent_ across all datasets. The hyperparameters are only different when a different model is used.

For more detailed responses, including experiment results, please refer to the original rebuttal.

We hope this summary addresses the key points of your review. Should there be any further details you wish to discuss or additional clarifications needed, please do not hesitate to reach out. We look forward to your feedback and are hopeful for a positive consideration.

Sincerely,

Authors
Dear Reviewer tU6b,

Thank you for your time and insightful feedback. Especially, we would like to _thank you_ for the suggestion to include a _direct evaluation of translation quality (W2)_. In response, we have added new experiments to measure the distributional differences between real and translated target LiDAR with MMD and JSD metrics. We believe this suggestion is genuinely valuable for improving the compellingness of our work.

Here, we would like to offer an additional summarized version of our responses for your reference:

- **W1. Motivation of the Work**
  - The main question raised is when we can already "accurately model the target domain data", why do we need to "translate the source domain data into the target domain"? The short answer is that we first model the "target LiDAR pattern" with a minimum dataset, and with this, we translate the "source data" to the "target data". There are two concepts here: what we are modeling is the **target domain LiDAR pattern**, but what we are translating is the **source domain data**. The goal is to effectively utilize LiDAR data collected from various source domain LiDAR sensors to jointly train a model that can be effectively used on the target domain, enabling data scaling up.
- **W2. Direct Evaluation for Translation**
  - We add new experiments (result table in original rebuttal W2) to directly measure the translation quality with Maximum-Mean Discrepancy (MMD) and Jensen–Shannon divergence (JSD). The results show that the translated LiDAR data closely matches the target domain data distribution, validating the effectiveness of our translation method.
- **W3. Comparison with Other Methods**
  - We provide comparisons with the current state-of-the-art data-based domain adaptation method, ReSimAD (ICLR24). The results show that our method outperforms ReSimAD in various domain adaptation tasks.
- **Q1. Ensuring Accuracy of Target Data Modeling**
  - See the **Updated Response to Q1** in the previous official comment.
- **Q2. Cost of LiDAR Modeling**
  - We agree with the reviewer that "the objective of domain adaptation is to address the challenge of having limited data or labels in the target domain".
  - We add a new experiment (results attached in original rebuttal) to show that only with very limited (1-4 target domain scenes without annotation), LiT can effectively model the target domain LiDAR pattern and achieve good performance. We provide a detailed analysis of the runtime cost and performance of LiDAR modeling. This shows that the cost of LiDAR modeling is quite small.

For more detailed responses, please refer to the original rebuttal and the updated response to Q1 outlined earlier.

We hope our response and additional experiments have addressed your concerns. If you have further questions or need additional clarification, please let us know and we are more than happy to engage in further discussions. We appreciate your feedback and are hopeful for a positive consideration.

Sincerely,

Authors
Dear Reviewer tU6b,

Thank you for your time and all of the insightful feedback. We would like to offer an **updated response for Q1**.

### Q1. How can we ensure the accuracy of modeling the target data? Will the differences between simulated data and real data have negative impacts?

- **Q1.1 How can we ensure the accuracy of modeling the target data?**

  - **Direct validation of the source -> target translation**: Thanks to your suggestions, we have added a new experiment to evaluate the distributional differences between the translated data and the target data. We follow LiDARDM and UltraLiDAR to evaluate the distribution differences with Maximum-Mean Discrepancy (MMD) and Jensen–Shannon divergence (JSD). We show that the translated data is closer to the target domain data compared to the source domain data.

    - Waymo -> KITTI
      | Input style | GT style | MMD (↓) | JSD (↓) |
      | ------------------------- | -------- | ------------- | --------- |
      | Waymo (baseline) | KITTI | 8.817e-04 | 0.273 |
      | Waymo translated to KITTI (ours) | KITTI | **3.268e-04** | **0.180** |
    - Waymo -> nuScenes
      | Input style | GT style | MMD (↓) | JSD (↓) |
      | ------------------------- | -------- | ------------- | --------- |
      | Waymo (baseline) | nuScenes | 2.310e-03 | 0.380 |
      | Waymo translated to nuScenes (ours) | nuScenes | **6.583e-04** | **0.205** |
    - nuScenes -> KITTI
      | Input style | GT style | MMD (↓) | JSD (↓) |
      | ------------------------- | -------- | ------------- | --------- |
      | nuScenes (baseline) | KITTI | 8.725e-04 | 0.220 |
      | nuScenes translated to KITTI (ours) | KITTI | **2.107e-04** | **0.164** |

  - **Verification through downstream tasks**: We evaluate the effectiveness of LiT translator through downstream detection tasks, where we train a model with the source domain and test it on the target domain. The results show that our method outperforms previous state-of-the-art methods (Baseline, SN, ST3D, and ReSimAD) in all translation cases. The results are provided in Table 2 and Table 3 of the main paper.

- **Q1.2 Will the differences between simulated data and real data have negative impacts?**

  Real target domain data with annotation, if available, is typically better than the simulated data. However, annotated real target domain data can be costly to obtain. LiT's main goal is to address the scenario _where real annotated target domain data is limited_. More importantly, LiT enables scaling up with simulated data from different source domains, and can even surpass real target domain training in some metrics with purely translated data.

  We provide an experiment to study the impact of using "real target domain data versus translated target domain data". In particular, we set KITTI as the target domain, and we use different combinations of real and translated data for training the SECOND-IOU model. The results are summarized below:
  | | Training set | Test set | Real or translated training data? | AP_BEV (↑) | AP_3D (↑) |
  | --- | ------------------------ | -------- | --------------------------------- | ---------- | --------- |
  | (a) | Waymo without translation | KITTI | No translation | 67.64 | 27.48 |
  | (b) | Waymo | KITTI | Purely translated | 82.55 | 69.94 |
  | (c) | Waymo + nuScenes | KITTI | Purely translated | **84.45** | 71.58 |
  | (d) | Waymo + nuScenes + KITTI | KITTI | Mix of translated and real | **87.52** | **75.76** |
  | (e) | KITTI | KITTI | Purely real | 83.29 | 73.45 |

  (Result (a) is from Table 1. Results (b) to (e) are from Table 3 of the main paper. Results better than training on "purely real" target set are highlighted in bold.)

  We can conclude that:

  - With translation is significantly better than no translation, as we see that (b) is significantly better than (a).
  - Training with real target data only (d) can be better than training with one translated source domain data (b). This shows the difference between real and translated data.
  - However, if we add more source domain data in experiment (c), the performance improves, although (c) has never seen the real target domain data during training, it does better than (e) in AP_BEV already. This shows that with LiT translation, we are able to scale up the training data to improve the performance, even surpassing training on real target domain data.
  - When we have both translated and real target domain data in training (d), the performance is the best, outperforming the purely real target domain data training (e).

  Therefore, the differences between simulated data and real data do have an impact, but with carefully designed translation strategies, we show that with purely translated data, it is possible to scale up the simulated training set and surpass real data training in some metrics (AP_BEV in experiment (c) is better than (e)); and when we have both translated and real data, the performance is the best (d).

Sincerely,

Authors
### Updates on W2. Inclusion of newer detection models

> It's advisable to conduct experiments with more recent models such as CenterPoint and other SOTA models to further demonstrate the effect of domain unification on SOTA models and potentially achieve new SOTA results.

Dear Reviewer omtq,

We sincerely thank you for your constructive suggestion to evaluate recent models in W2. We would like to provide an update regarding integrating LiT with the **CenterPoint** model. We perform the Waymo -> KITTI translation tasks, comparing the performance across various setups: baseline (no translation), ST3D (model-based adaptation), our method (LiT, direct translation), and an oracle (direct training on KITTI). The results are summarized below:

| **CenterPoint**                        | **AP_BEV (↑)** | **AP_3D (↑)** |
|----------------------------------------|----------------|---------------|
| Source only (Baseline)                 | 75.26          | 45.46         |
| ST3D (Model-based adaptation)          | 76.97          | 49.72         |
| **LiT (Ours, Direct translation)**     | **77.86**      | **62.67**     |
| Oracle (Direct training with target)   | 81.23          | 71.24         |

These results illustrate that our LiT method surpasses both the baseline and ST3D in the AP_BEV and AP_3D metrics, with a particularly significant boost in AP_3D, showcasing the practical benefits of our approach in domain translation. The performance of LiT closely approaches that of the oracle, highlighting our method's potential to effectively bridge the domain gap. 

Thank you again for your valuable feedback. We plan to conduct additional experiments on different translation tasks and evaluate other SOTA models with LiT for the revised manuscript. Please let us know if you have any further suggestions or questions.

Sincerely,

Authors
All my concerns are well addressed.