ReSimAD: Zero-Shot 3D Domain Transfer for Autonomous Driving with Source Reconstruction and Target Simulation
Authors: Bo Zhang
Xinyu Cai
Jiakang Yuan
Donglin Yang
Jianfei Guo
Xiangchao Yan
Renqiu Xia
Botian Shi
Min Dou
Tao Chen
Si Liu
Junchi Yan
Yu Qiao
Venue: ICLR 2024 poster (2024)
Keywords: Autonomous Driving
3D Domain Transfer
Zero-shot 3D Detection
Primary area: applications to robotics, autonomy, planning

Abstract: Domain shifts such as sensor type changes and geographical situation variations are prevalent in Autonomous Driving (AD), which poses a challenge since AD model relying on the previous domain knowledge can be hardly directly deployed to a new domain without additional costs. In this paper, we provide a new perspective and approach of alleviating the domain shifts, by proposing a Reconstruction-Simulation-Perception (ReSimAD) scheme. Specifically, the implicit reconstruction process is based on the knowledge from the previous old domain, aiming to convert the domain-related knowledge into domain-invariant representations, e.g., 3D scene-level meshes. Besides, the point clouds simulation process of multiple new domains is conditioned on the above reconstructed 3D meshes, where the target-domain-like simulation samples can be obtained, thus reducing the cost of collecting and annotating new-domain data for the subsequent perception process. For experiments, we consider different cross-domain situations such as Waymo-to-KITTI, Waymo-to-nuScenes, etc, to verify the zero-shot target-domain perception using ReSimAD. Results demonstrate that our method is beneficial to boost the domain generalization ability, even promising for 3D pre-training. Code and simulated points are available at: https://github.com/PJLab-ADG/3DTrans

Published as a conference paper at ICLR 2024
RESIMAD: ZERO-SHOT 3D DOMAIN TRANSFER FOR AUTONOMOUS DRIVING WITH SOURCE RECON- STRUCTION AND TARGET SIMULATION
Bo Zhang'.., Xinyu Cait.*a, Jiakang Yuan Donglin Yang", Jianfei Guo', Xiangchao Yan', Renqiu Xia Botian Shil-t, Min Dou', Tao Chen', Si Liu", Junchi 2 Yu Qiao'
1 Shanghai Artificial Intelligence Laboratory, 2 Shanghai Jiao Tong University
3 Fudan University, 4 Beihang University
Equal Contribution, Corresponding Authors, Project Leader
ABSTRACT
Domain shifts such as sensor type changes and geographical situation variations are prevalent in Autonomous Driving (AD), which poses a challenge since AD model relying on the previous domain knowledge can be hardly directly de- ployed a new domain without additional costs. this paper, we provide a new perspective and approach of alleviating the domain shifts, by propos- ing a Reconstruction-Simulall Simulation-Perception (ReSimAD) scheme. Specifically, the implicit reconstruction process is based on the knowledge from the previ- ous old domain, aiming to convert the domain-related knowledge into domain- invariant representations, e.g., 3D scene-level meshes. Besides, the point clouds simulation process of multiple new domains is conditioned on the above recon- structed 3D meshes, where the target-domain-like simulation samples can be obtained, thus reducing the cost of collecting and annotating new-domain data for the subsequent perception process. For experiments, we consider different cross-domain situations such as Waymo-to-KITTL, Waymo-to-nuSeenns, etc, to verify the zero-shot target-domain perception using ReSimAD. Results demon- strate that our method is beneficial to boost the domain generalization ability. even promising for 3D pre-training. Code and simulated points are available at: https: / thub. com / /t..//Jaabb---.. ADG/ CDDG//DTrass..
INTRODUCTION
Autonomous Driving (AD) aims to perform the perception of the ego-car's surroundings and further make decisions without human involvement (Sun et al., 2020; Li et al., 2023). In recent years, an increasing number of self- self-driving vehicles the L2 level are gradually entering lives, becoming an indispensable part of traffic elements (Mozaffari et al., 2020). Although it has achieved promising progress, there are still many issues that need be addressed, such continuous data collection and annotation efforts, before accomplishing robust autonomous driving system.
Considering a more demanding yet practical cross-domain scenario: As manufacturer of AD sys- tem, the self-driving vehicle product is required to be updated to the next version for business pur- poses. However, this product update process may encounter the following challenges caused by domain shifts. The base model of the current-version self-driving vehicle is well-trained on the massive labeled source domain collected on the previous old domain knowledge (such as previous sensor technologies or data acquisition cities), but it needs to achieve good performance on differ- ent domains such as next-generation sensor technology or unseen streets. Unfortunately, previous works (Yang et al., 2022; Wei et al., 2022; Yuan et al., 2023b; Huang et al., 2023; Xu et al., 2021a; Geiger et al., 2012; Wang et al., 2020; Dong et al., 2023) pointed out that the existing AD models, including 3D detection (Shi et al., 2021: Deng al., 2021), segmentation (Hou al., 2022; Graham et al., 2018), motion (Shi et al., 2022), and planning models (Teng et al., 2023), typically face serious performance drop the target domain with such cross-domain shifts.
One commonly used solution to transfer the base model from its original domain to a new target domain is the supervised fine-tuning the new target domain (Yin et al., 2021; Yuan et al., 2023a),

---

Published as a conference paper at ICLR 2024
Original LiDAR sensor Target
New seneor
(a) simulate
W Nayoo to cemes Performance
Domain
perameter
Simulaton CA ALRA Scenes
Simulator
Points of of Original Denain
Model
da) collect
C Data Collection
(3)
New secsor
Cost
Diversity
New sensor
(a)
Low Low
parametee
(c simulate
(b)
High High
Reconstructed Mesh
Simulator
Low
High
Different paradigms for cross-domain Autonomous Driving (AD), where Rec. de- notes the employed reconstruction scheme. (a) By directly simulating target-domain data from CARLA (Dosovitskiy et al., 2017), expands the number of training samples from the target do- main. But the diversity of the simulated data directly from CARLA is low. (b) Other works (Yang al., 2021; 2022; Yuan et al., 2023b; Wei et al., 2022) often employ Unsupervised Domain Adapta- tion (UDA) to enable learning from target-domain data distribution. But this process needs to collect massive target-domain samples from the real world, which expensive.
Figure 1:
(c) In the proposed ReSi- mAD paradigm, the well-labeled data from the old domain is utilized to reconstruct the 3D scene, and then, the target-domain like data is simulated from the reconstructed scene.
which is expensive due to that it needs to pay a high price for data acquisition and human an- notation of the target domain. Besides, the long development cycle of data acquisition and human annotation on a new domain (e.g., next-generation sensor for self-driving vehicles) could give rise the delay in the product delivery of the next-generation self-driving vehicle.
Recently, researchers have tried to bridge the domain gaps, by either rendering autonomous driving scenes directly from the simulation engine (Dosovitskiy al., 2017) reduce the data acquisition cost, or leveraging Unsupervised Domain Adaptation (UDA) technique (Yang et al., 2021; Wei et al., 2022: Yuan al., 2023b; Zhang et al., 2023; Xu al., 2021a; Yang al., 2022) to reduce human annotation cost, which is illustrated in Fig.
Different from the above-mentioned research works, we consider the particular issue faced by the AD manufacturer: How to achieve a zero-shot source-to-targed model transfer. Note that the zero- shot means that for the target domain, there is almost no additional data acquisition and labeling cost. To achieve this goal, we propose ReSimAD a unified ceccptiinnsimulationnn- perceppioonnnn pipeline. Given the access to the annotations from the source domain, we propose to leverage the source- domain annotated data to reconstruct the 3D real scene, producing the 3D mesh to decouple the domain characteristics. Further, the 3D mesh can be fed into the simulation engine to re-inject some target comainn-reltted characteristics such as LiDAR sensor parameter setting, according to minor prior information that we have obtained about the target domain.
Empirically. we obtain the turget-domain like points by the wwwpnnsttrcttionnsmaatttttt scheme, show- ing the effectiveness of the simulated points on the real target-domain scenario. We conduct exper- iments under Waymo-to-KIIin show excellent zero-shot ability when the perception model faces the domain shift, even surpassing some UDA baselines that have access to extensive real-world target domain data.
Waymo-to-nuScenes, Waymo-to-ONCE: settings, and the results
Contribution. (1) We provide the autonomous driving community with the knowledge that, the scheme of source-domain reconstruction followed by target-domain simulation essentially improves the robustness for an unseen target domain. (2) We propose ReSimAD, a unified reconstruction-
simulation iimulation-perccptioon paradigm that addresses the domain shift issue, where the reconstructed 3D meshes decouple the domain characteristics, bridging the well-labeled old domain and an unseen new domain. (3) By extensive experiments on different datasets with distinct domain shifts, ReSi- mAD achieves high 3D detection accuracy under zero-shot target-domain setting, even outperform- ing the unsupervised domain adaptation method that has to access massive real target domain data.
RELATED WORKS
3D Object Detection under Different Domains. The domain variations make the model trained on the lableelabeled source domain hard to achieve a satisfactory detection accuracy on the tar- get domain with domain differences. Notably, the idea of Domain Adaptation (DA) commu-
2

---

Published as a conference paper at ICLR 2024
(a) Real domain
(b) Simulated domain
Figure 2: Visualization results between (a) real domain and (b) simulated domain. The domain simulated by ReSimAD is close to the real domain, such as slope on the road.
nity (Long et al., 2015; Fu et al., 2021) has been well developed, especially for the 3D cross-domain works (Yuan et al., 2023b; Zhang et al., 2023; Yang et al., 2021; Wei et al., 2022; Xu et al., 2021a) that handle the domain shifts on the Autonomous Driving (AD) scene. For example, ST3D (Yang et al., 2021) and I. IIDAR-Distillttion (Wei el., 2022) design a self-training pseudo-labeling pipeline and beam distillation method, respectively, assuming that the real data from the target domain are available for the training process. Besides, Bi3D (Yuan et al., 2023b) proposes to actively sample both the source and target samples to reduce the domain gap. Recently Uni3D (Zhang et al., 2023) extends the study of cross-domain adaptation, investigating the possibility of joint training on multi- ple merged point datasets. Different from previous DA study works, we address the domain gaps from another perspective: 1) domain- decoupling based on the old-domain data, and 2) re-sampling new-domain-like data based on a simulator in a cost-free manner.
Scene Reconstruction and Simulation. Environment simulation (Dosovitskiy et al., 2017) is of- ten used AD scenes to reduce the costs of collecting expensive training data. Early works (Cai et al., 2023; Dosovitskiy et al., 2017: Gschwandtner et al., 2011) mainly focus on leveraging the vir- tual sensor simulation such as CARLA (Dosovitskiy et al., 2017) or Blensor (Gschwandtner et al., 2011). For example, by utilizing virtual sensor that can be configured to simulate real-world de- vices, additional synthetic data can be generated for model training (Wang et al., 2019). Recently, researchers have started to real data to strengthen the realism of the simulation process under the complex world. For example, LiDARsim (Manivasagam et al., 2020) builds an asset Library of 3D static maps and 3D dynamic objects, by means of driving around several cities in the real world. UniSim (Yang et al., 2023) proposes a closed-loop multi-sensor simulation scheme, which reconstructs the dynamic objects and static background, further producing different observations of new viewpoint. Different from these scene reconstruction and simulation methods, our ReSi- mAD fully investigates how to boost the zero-shot target-domain perception performance using the reconstruction-siminee simulatiin pipeline, where we separate the reconstruction and simulation process, allowing the model to simulate wider data distribution variations.
THE CONOONSTRRUTIIONSSMMMLIIII
DATASET
To compare with recent research works (Yang al., 2021; Yuan al., 2023b) in Domain Adapta- tion (DA) community that aims to study the cross-domain adaptability of the detection models, we introduce the first 3D econstraction-simulaaii sequences (Sun et al., 2020), with different sensor settings.
dataset, which is constructed based on the Waymo
We follow the DA community and also employ Waymo (Sun al., 2020) dataset as the source (old) domain and other datasets, e.g. nuScenes (Caesar et al., 2020) and KITTI (Geiger et al., 2012), as the target (new) domain. As result, the implicit reconstruction performed on Waymo in order to generate the 3D scene-level meshes, while we simulate KITTI, nuScenes, and ONCE scenarios according to the Waymo-based 3D meshes. The detailed implementation is elaborated in Sec. 4. Besides, Waymo sensor features one top LiDAR and four side LiDARs (Sun et al., 2020), which facilitate a broader longitudinal perception range capable of encompassing the narrow longitudinal field of view of other datasets such as KITTI (Geiger et al., 2012).
3D Reconstructed Meshes and Simulated Points. We generate scene-level 3D meshes from Waymo dataset. Note that during the evaluation stage of reconstructed 3D meshes, we can perform the re-raycasting according to Eq. 3 on Waymo domain, and select 146 meshes with the highest re- construction scores calculated using the Chamfer Distance (CD) distance between the simulated and real points. For cross dooss-t setting, we simulate approximately 26K ~ 29K samples per domain. The visualization results of the simulation data are illustrated in Fig. 2 and Appendix A.
caannstrretion-SSmmaatiti Dataset Analyses. Considering that the generated point clouds not only need to boost the model performance in the target domain, but also valuable for 3D pre-training
3

---

Published as a conference paper at ICLR 2024
Collected old domain point clouds
Unreal engine render
LiDARSim ray casi
Simulated data
Detector
Train
PV BCNN
P--CCNNN
dynamic object seqmentation
Differ ent scenarios
Unlabeled roal data
Apply with
Mulii stitching
Simulated new domain point clouds
Zero-shot Inference
Traflic scenario reconstructin
LIDAR-onh Implicit reconstruction
(a) Point-to-Mesi Reconstruction
(b) Mesh-to-Poin Simulation
Zero-shot Perception
Figure 3: The overview of ReSimAD, which consists of point-to-mesh reconstruction, mesh-to- point simulation, and zero-shot perception. Each part is detailed in Sec.
to enhance the backbone feature generalization ability, we simulate the target-domain points with more diversity of object size. The distribution of object size the four simulated domains cttesseeettatiioo----m wide object size distribution compared with the off-the-shelf public dataset such ONCE (Mao et al., 2021), 1), Waymo (Sun et al., 2020).
shown in Fig. 4. It can be seen from Fig 4 that the reconstraciion-iiulaato dataset covers
RESIMAD: RECONSTRUCTION, SIMULATION, PERCEPTION PIPELINE
ReSimAD Overview. As illustrated in Fig. 3, ReSimAD covers three steps.
I) Point-to-mesh Implicit Reconstruction: This step aims to map points from the source domain to implicit geometry. To obtain highly consistent simulation data conforming to the distribution of target-domain LiDAR, we reconstruct the real-yet-diverse street scene background, dynamic traffic flow information using LiDAR-only reconstruction (Guo et al., 2023)*. We select Waymo as the source domain for scenario reconstruction.
2) Mesh-to-point Rendering:
The purpose of this step is to simulate the lurget-domain-lin points given the reconstructed implicit geometry, by changing the LiDAR-sensor and scene layout. Specif- ically, we employ PCSim (Cai et al., 2023) and reproduce the sensor configuration scheme used in the target domain, including LiDAR scan modes, physical characteristics, and deployment locations.
The well-simulated points are fed into the perception module that can help the original model enhance the cross-domain generalization for common domain variations such changes in LiDAR types.
3) Zero-shot Perception Process:
!) Point-to-mesh Implicit Reconstruction. Inspired by previous works such as StreetSurf, DeepSDF and NeuS (Guo al., 2023; Park et al., 2019; Wang al., 2021), utilize the Implicit Neural Reconstruction method. By leveraging neural networks to encode signed distance functions (Park al., 2019; Wang al., 2021; Oechsle et al., 2021; Zhu al., 2022), we achieve the synthesis of high-quality 3D models with exceptional resolution and efficient memory usage.
Different from recent methods (Deng et al., 2022; Rematas et al., 2022) that use RGB images to refine implicit representation, we carry out LiDAR-only setting, which takes sparse LiDAR point clouds as input and generates an implicit Signed Distance Field (SDF) field. LiDAR-based Implicit Neural Reconstruction (LINR) allows to be unaffected by lighting conditions, e.g. weak lighting and changing weather in practical scenes, and obtain a richer and larger dataset of the 3D scene.
For ray r(o, d) emitted from the origin R with the direction E R3, apply volume rendering for LiDAR to train the SDF network, the rendered depth D can be formulated as:
k
D(r) Σ Nat,,
'Code of implicit reconstruction is available at: https: http:///gghhthtco/////////////// "Code of mesh-to-poots point rendering available https: //gi thub. com com/Pullass ADG/PCSim
4

---

Published as a conference paper at ICLR 2024
where t, is the depth of the i-th sampling point, T, I[ tance, and 0; is obtained by employing the close-range model in NeuS (Wang et al., 2021).
(1 aa) is the accumulated transmit-
Drawing inspiration from StreetSurf (Guo al., 2023), the reconstruction input is derived from LiDAR rays, and the output is the predicted depth. On each sampled LiDAR beam Plidar, we apply logarithm L.1 loss Dicrddy) rendered depth of the combined close-range and distant-view model:
Ligaametry In (Blorddv) (Triaar
D (Fiidar)
+ 1)
However, the LINR method still faces some challenges. A single LiDAR point cloud frame captures only a portion of the comprehensive information contained within a standard RGB image, due to the inherent sparsity of LiDAR data. This disparity underscores the potential inadequacy of depth rendering in providing the necessary geometric details for effective training. Consequently, this could lead to the generation of substantial volume of artifacts within the resulting reconstructed mesh. To tackle this challenge, we consolidate all frames from the corresponding sequence within Waymo dataset (Sun et al., 2020). Please refer to Appendix A. for more details of point cloud registration when consolidating all frames. Next, use point neighborhood statistics to filter out the outlier point clouds for each scenario.
Considering the constraint of top LiDAR's vertical field of view in Waymo dataset, obtaining point clouds only between -17.6 and 2.4 imposes limitations on the reconstruction tall surrounding buildings. To tackle this challenge, we introduce solution that incorporates point clouds from side LiDAR (blind compensation) into the sampling sequence. Four side LiDARs are strategically positioned the front, rear, and sides of the vehicle, with vertical field of view spanning from -90º to 30. This effectively compensates for the inadequate field of view of the top LiDAR. Since the disparity in point cloud density between the side LiDAR and top LiDAR, we opt assign higher sampling weights to the side LiDAR to enhance the quality of scene reconstruction for tall buildings.
After reconstructing the implicit surface, we can obtain continuous representations of scene ge- ometry with finer granularity, facilitating the extraction of high-resolution meshes for subsequent rendering in a selected simulator. For more details on the training settings of LINR, please refer to StreetSurf (Guo et al., 2023).
Reconstruction Evaluation. Due to the occlusion caused by dynamic objects and the influence of LiDAR noise, implicit representations might be lower than expected, posing challenges for cross- domain adaptation. Hence, we conduct an evaluation for the reconstruction accuracy. Since we can access the real-world point clouds of the old domain, we evaluate the accuracy of the reconstruction process by re-raycasting the point clouds for the old domain.
We use a set of metrics for the reconstruction accuracy between the rendered points & and original collected LiDAR points G, with Root Mean Square Error (RMSE) and Chamfer Distance (CD):
CD( =
1 min ||x - - + |G|
Σ
min lly XEG
x I
(3)
G
XEG
YEG
yeG
where the reconstructed scores and some detailed descriptions are shown in Table 6 in Appendix.
2) Mesh-to-point Rendering. After obtaining the static background mesh through the above- mentioned LINR, we use the Blender Python API to convert the mesh data from .ply format to 3D model files using ffx format, and finally load the background mesh as an asset into CARLA (Dosovitskiy et al., 2017), an open-source simulator for autonomous driving research.
For the appearance match of traffic participants, we obtain the categories and the three-dimensional size of the bounding boxes in each frame of data, through the annotations from Waymo (Sun al., 2020). According to this information, we search for the digital asset with the closest size among the traffic participants of the same category in the digital asset library of CARLA (Dosovitskiy al., 2017), and use it as the model of the traffic participant. According to the scene truth information available in the CARLA simulator, we developed a bounding box extraction tool for each de- tectable target in the traffic scene, and converted it into the label format of the target domain (such as KITTI (Geiger et al., 2012)).
5

---

Published as a conference paper at ICLR 2024
It can be observed from Fig. 4 that the distribu- tion of object sizes is different across datasets. To ensure the consistency between the simula- tion dataset and the general vehicle size in the target domain,
6.2
2.5
first perform function map-
ping on the size of each traffic participant ac- cording the statistical results, and then com- plete the assets matching process.
2.2
2
For motion simulation of traffic participants, we sort out the traffic scene coordinate system and update the location and pose dynamic objects frame by frame. For each segment, we take the grounding point of the ego vehicle center in the first frame as the coordinate origin. The 6D pose of the ego vehicle is updated via the dif- ference between ego vehicle labels in different frames. Other dynamic targets are updated by the relative 6D pose of the ego vehicle in the la- bel information of each frame. The 6D pose of simulated object pe in the t-th frame can be represented as (I, roll, yaw, pitch) in the simulator. The update of the ego pose and dynamic object pose is:
1.6
0.5
Length
Height
KITTI
Waymo
Figure 4: Distribution differences of object size (Length, Width, and Height) across datasets. Compared with the off-the-shelf public datasets, the simulation dataset constructed by the proposed ReSimAD covers wider distribution.
Pt ego
L - egoo
Piarget = target + Pego:
(4)
where I. denotes the absolute pose in Waymo world coordinate system, and L' is the relative pose in the coordinate system relative to the ego pose.
To study the effects of traffic scenario reconstruction and LiDAR simulation synthetic data thenticity and zero-shot domain adaptation performance, three datasets are constructed. In addition the implicit reconstruction simulation dataset according to the aforementioned method, the sensor- like LiDAR simulation dataset and the default LiDAR simulation dataset both based on the CARLA scenario background are also constructed, by the OpenCDA tool (Xu et al., 2021b).
The main difference between the sensor-like LiDAR dataset and the default LiDAR simulation dataset that, the number of LiDAR channels and the vertical field of view are different. The default LiDAR has a fixed 32-channel configuration with a vertical field of view ranging from -30 to 10 degrees, and beams are uniformly distributed. Meanwhile, leveraging the simulation library (Cai et al., 2023), the features of the sensor-like LiDAR are equal to those of the corresponding sensor setting from the target domain. The detection range, points emitted per second, rotation frequency, and drop-rate of the target domain LiDAR also nearly identical to those of the default LiDAR. Due to the need for vehicle traffic flow to match the road network structure, for data simulation based on the CARLA static background, completed the matching of vehicle traffic flow.
3) Zero-shot Perception Process. To further achieve the closed-loop simulation verification, we use domain, and perform an evaluation
the simulated points Xsim to train baseline model on the on the real samples Xreat of the validation set from the new domain. Specifically, we verify our method the 3D detection task, and the overall loss Itrain and evaluation process Eeont are:
(5) where Lets and Greg are the class and regression loss of the detection head, respectively. &reat is the prediction. AP is the metric score between the model prediction and ground truth (see Sec. 5.
Ltrain Lets (Xamm) + Creg Exreg(Xsin)) (Xsim), Egont = AP(X X XGT
5 EXPERIMENTS
5.1 EXPERIMENTAL. SETUP
Dataset Selection for Experiments. In order to make fair comparisons with the existing UDA methods (Yang et al., 2021; 2022; Wei et al., 2022; Yuan et al., 2023b), we align their cross-dataset setting, and employ the Waymo-to-KIn KITTI. Waymo-to-nuSee nuSeenes setting to study the model cross- domain ability under the 3D scenario. Besides, to achieve high-quality reconstruction results,

---

Published as a conference paper at ICLR 2024
Table I: Results on different adaptation scenarios under zero-shot, Unsupervised Domain Adap- tation (UDA), and Fully-supervised Training (FT) settings. Following (Yang et al., 2021), we re- port APBEY and AP over 40 positions' recall for car category. Source Only denotes that the pre-trained detector is directly evaluated on the target domain, and Oracle represents the detection results trained using the fully-annotated target domain. Closed Gap denotes the detection accuracy gap closed by various methods along Source Only and Oracle results. Our method belongs zero- shot, while UDA assumes that all REAL target samples are available for model training. The best results of model adaptation are marked in bold.
Tisk
Method
Setting
APray/ AF
PVRCNN
Closed Gap
AFazy / /AP
PV PVNNEN++
Closed Gap
Source Only
Zem--doo
61,18/210
67.68 2088
ST3D (Yang
UDA
$4.00 64.78
+52.45% 700.71%%
33.166 / 76.53%
Woymo-KITTI
CARLA-defaaa (Dovovisacy 2017
ReSimAD
Zero-abot
7em-shor
81.01/ 55.42
5KG97 34
2009% 4230
19%
82,06
5056873155
-126715/7
+10K-99%
32%
Sensor-fi like (Manivasggom al., 2020)
Zem-shot
71,09/400
+33.64%//431.000
54.16/34116
-108.03%
043600066633000 58%
Okapie
FT
11.1277003
Source Unly
Zero-ubot
31,02/1 17.75
29:937 18.77
ST3D (Ying of al, 2021)
UDA
36.42/22.99 22.99
444 /-25 18%
34.68/
17.17
+19.405 7-7.028
Waymo-nnSSenne
CARLA-defall (Dnsmillltty al, 201
ReSimAD (Ouns)
Zeno sbot
Zem-dbox
24.147 12.26
37.85/ 21.33
111. 2-26677766666 38%
00.92%/-11.2000 20%
22.147 7.04
40.73/23.72
-31,8224 7:30%
/-11122/-21.526
Scusor like (Manivasagum sl. 2020)
Zero-bhot
28.90/ 15.25
-9.59% -11.53%
35.98/ 19.52
+24.71% 763.96%
Oracle
53.117 38.56
54,417
Only
50.06
68.727 54
ST3D (Yang al., 2021)
68.13/41.5
-3.43% +5.0%%
+5.34%
Waymo-+ONCE
C'ALAA defouik (Dosovitslay et al, 2017]
ReSimAD (Ours)
Zero-boot
Zero-boot
70.97/
58.22 52.11
+10.68%-7
.35 --1.999999
30%
74.52/ 53.91
56,80 $2.77
+29
.61.29% 7-3.00%
434
Scnsor like (Manivasagum et 2020)
Zeno abot
61.36/ 35.77
-36.96% / 8804%
69.13 44.16
+2.111% / 7211.44
Oracke
FT
88.9577955
88.17/ 79.34
have to merge multiple frames sampled from the same 3D sequence perform the reconstruction process, which is also inspired by StreetSurf (Guo et al., 2023).
Implementation. We first train the base model on the labeled source domain and evaluate the cross-domain performance of the trained source model on the target domain. We define the domain variations in different cases including cross-region, cross-beam, and cross-dataset, etc. For training on the simulated data, we use Adam optimizer with a one-cycle learning rate decay schedule, where we set the maximum learning rate to 0.002. The weight decay is set to 0.01. The total simulation data-related training process ends when 10 epochs are reached using 4 NVIDIA Tesla A100 GPUs. Evaluation Metric. Following previous cross-domain research works (Yang al., 2021; 2022; Wei et al., 2022; Yuan et al., 2023b; Zhang et al., 2023), we use AP for evaluation in both the Bird's Eye View (BEV) and 3D over 40 recall positions, where the IoU threshold is set to 0.7.
Comparison Baselines. We compare the proposed ReSimAD with three typical baselines of allevi- ating cross-domain shifts: 1) Data simulation-related baseline using simulation engine (Dosovitskiy et al., 2017); 2) Sensor simulation-related baseline by changing the sensor parameter setting; 3) Label-efficient baseline (Yang et al., 2021).
1) CARLA-default: We use the CARLA (Dosovitskiy et al., 2017) simulation engine to generate the simulated data. For this baseline, foreground objects are added into the simulated scenario by searching from CARLA to find the closest object-size digital asset, as described in Sec. 4.
2) Sensor-like: We assume that the target-domain sensor setting can be obtained, and thus, we also change the sensor parameter setting in CARLA and simulate point data so that the LiDAR-beam distribution is similar to the target domain scene. Note that the above two baseline settings only produce more simulated data that conform to the target-domain distribution, and we directly use the simulated data to fine-tune our base models and observe their performance in the target domain.
3) ST3D: We compare with ST3D (Yang et al., 2021), a popular Unsupervised Domain Adaptation (UDA) technique reducing the cross-domain differences of point clouds in a label-officient manner.
5.2
CROSS-DOMAIN EXPERIMENTS
5.2.1 RESIMAD BOOSTS ZERO-SHOT 3D OBJECT DETECTION
To ensure the fairnsss of experiments, we first compare the proposed ReSimAD with the data simulation-related baselines: CARLA-default (Dosovitskiy et al., 2017) and Sensor-like (Mani- vasagam et al., 2020). It can be observed from Table 1 that, our ReSimAD achieves the best zero- shot 3D detection accuracy for all cross-domain settings both PV-RCNN (Shi et al., 2020) and PV-RCNN+ (Shi et al., 2021). Also, we found that the Sensor-like baseline (Manivasagam et al., 2020) has stronger robustness against domain differences compared with CARLA-default (Doso-
7

---

Published as a conference paper at ICLR 2024
CARLA-delaul
CARLA CARLA-default
ReSimAD (a) nuSoenes
ReSimAD
(b) KITTI
Figure 5: Visualization of simulated points using CARLA default background (first row) or recon-
structed one (second row) from the real scenarios. The last row: reconstructed background.
vitskiy et al., 2017), since we conduct sensor-level simulation in advance according to the LiDAR parameter settings of the target domain. However, due the differences background distribution between simulated and real scenes, it is still difficult to achieve satisfactory cross-domain perfor- mance using Sensor-like baseline (Manivasagam al., 2020) alone (i.e., only 71.09% 10009%/%488800 40.80% for Waymo-to- to-TTT setting).
Besides, Table compares Sensor-lit (Manivasagam et al., 2020) and ReSimAD, and it shows that ReSimAD can generally outperform the Sensor-like methods by around 5.98% 27. 27.49% under different types of cross-domain differences. Therefore, we believe that the authenticity of the point- cloud background distribution is also crucial for achieving zero-shot cross-dataset detection.
Table shows the results of leveraging Unsu- pervised Domain Adaptation (UDA) technique. The major difference between UDA and ReSi- mAD that, the former employs samples from real scenes of the target domain for model adap- tation, while the latter cannot access any real point cloud data from the target domain. From Table 1, the cross-domain results achieved by our ReSimAD are comparable to that achieved by UDA method (Yang et al., 2021). This result indicates that our method can greatly reduce the cost of data acquisition, and further, shorten the development cycle of model retraining when the LiDAR sensor needs to be upgraded.
Table Zero-shot and Fully-supervised (SFT and Oracle) results on the target domain. For SFT setting, we use the checkpoint pre-trained on the simulated data as the backbone initializa- tion, and fine-tune on the labeled target domain.
Sattng
Wayno- KIITI
Waymo-nnaee
PVAR
PV-RON (Shi
82.71
54.68/ /
PV-RCNN (S8i
55.09/911
/3.07/22 0000/38.0
PERCNN: 2021)
Zaneslec
PV-RCNN (Stá
67.95/81.55
30.94
RCNN+c 202))
56.39 /8.244
38.96
5.2.2 RESIMAD BOOSTS FULLLY-SUPERIISE 3D DETECTION
Another benefit of using data generated by ReSimAD is that a high-performance target-domain accuracy can obtained without accessing any target-domain real data distribution. We found that such a target-domain-like simulation process can further boost the Oracle results of the baseline.
Table 2 reports the results of using full human annotations from the target domain. Oracle represents the highest result achieved by the baseline model trained all labeled data from the target domain. SFT denotes that the network parameter of the bascline model is initialized by the weights trained from the simulation data. Table 2 shows that the backbone pre-trained using our simulated point clouds provides a better initialization for 3D detectors such as PV-RCNN+ and PV-RCNN.
5.3 3D PRE-TRAINING EXPERIMENTS
Overview of 3D Pre-training using Simulation Data. To verify ReSimAD can produce point data to help 3D pre-training, simulated point clouds using AD-PT (Yuan et al., 2023a), and then tne-tuned on the downstream real-world data. It saves lots of real-world data by using simulated data for 3D pre-training.
devise the following setting: 3D backbone is pre-trained the
8

---

Published as a conference paper at ICLR 2024
Table 3: The performance scalability over different amounts of training data, using the simulated dataset compared with the real dataset (ONCE), FT denotes the fine-tuning. Sim-data indicates the simulated point data using our method, and ONCE-data are collected in the
scene.
Pre-training Methods
Pre-training Data Source
Data Collection Cost
KITTI AFmay Al
Waymo
Overall
Car
Podestrian
Cyclist
AP
APH
From Scratch pre training)
Nome
None
70.57
84.50
57.06
70.14
69.97
67.58
AD-PT (Yunn al, 2023s)
55K Sim-data
None
84.53
57.02
71.65
67.87
AD-PT (Yaan al, 2023a)
110K Sim-data
None
71.74
84,82
58.30
72,10
70.46
68.42
(Yoan 2023a)
100K ONCE-data
High
73.01
84.75
71.55
69.23
(a) Reconstruction using VDBFusion
(b) Reconstruction using method
Figure 6: Visualization comparisons between the VDBFusion (Vizzo et al., 2022) and our recon-
struction method towards sequence on Waymo dataset.
Downstream Fine-tuning Results. We utilize ReSimAD to generate the data with more wider distribution point clouds. For fair comparison with their pre-training results in AD-PT (Yuan et al., 2023a), the target amount of simulation data generated by ReSimAD is approximately 100K. In Table 3, the baseline detector pre-trained on either simulated data or real-world data, and fine- tuned on both KITTI (Geiger al., 2012) and Waymo (Sun et al., 2020) benchmarks. Table 3 shows that, the performance of fine-tuning can be continuously increased using simulation pre-training data different scales. Overall, it achieves pre-training by leveraging different scales of simulated point clouds in zero-shot fashion.
5.4 FURTHER ANALYSES
Effectiveness of Reconstruction and Simulation. To verify the module- wise effectiveness of the proposed method, we visualize the point clouds rendered using different methods including CARLA simulator, and real-world reconstructed 3D scenario by our method in Fig. 5.
It shows that the simulated points obtained by ReSimAD cover more realistic scene informa- tion for the target domain, such road sur- face and streetscape. Fig. also visualizes the reconstructed mesh using different reconstruc- tion methods. Visualization results illustrate that compared with the VDBFusion (Vizzo et al., 2022), the implicitly reconstructed meshes by our ReSimAD show clear street view informa- tion and continuous geometric structure. Please refer to Appendix A.4 for more comparison re- sults.
Table 4:
Cross-dataset results using SECOND- IOU baseline, where #Sim-data denotes the num- ber of simulated target-domain samples.
Waymo-nauScon
Models
Setting
#Sim-clata
APpav AP
SECOND YOU
Zero-sho
0
24.57 15.12
SECOND-IOU
Zero-shot
5k
30.12 00.12/9.25
SECOND- IOU
Zero shot
10K
35.38 18.55
SECOOD-IOO
Zero-shot
36.45 18.94
SECOND-IOU
Oracle
0
50.54/ 33.41
Adaptability on Lightweight Baseline. Table 4 shows that the simulated point clouds are also effective for the lightweight baseline, which is more practical in real applications. We employ SECOND-IOU (Yan et al., 2018) as the baseline, and train it on the simulated data using ReSimAD. The results show that our method also obtains promising results on the one-stage 3D detector.
CONCLUSION
In this work, we study how to achieve a zero-shot domain transfer, and present ReSimAD consisting of a real-world point-level implicit reconstruction process and a mesh-to-point rendering process. We have conducted extensive experiments under zero-shot settings, and their results demonstrate the effectiveness of ReSimAD in producing target-domain-lie samples and achieving high target- domain perception ability, even helpful for 3D pre-training.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
The proposed ReSimAD tries to achieve zero-shot domain transfer from a new perspective: source- domain reconstruction and target-domain simulation. Such zero-shot method can significantly reduce the cost of both data acquisition and human annotation on the target domain. However, due that knowledge transfer from label-rich source domain an unseen target domain may contain deviations, the adapted 3D model may produce certain prediction errors towards the target domain. Thus, how to obtain target-domain model with high performance under the zero-shot setting needs to be further studied in the future.
REPRODUCIBILITY STATEMENT
The complete reconstruction simulation pipeline for ReSimAD has been included Sec. 4, and the Reconstruction-SSie Simullaiin Dataset introduced in Sec. 3 has been released. Besides, the source code of ReSimAD is available at: https: /ggttubb..tbt// www/ JLab ADG/3DTram
ACKNOWLEDGEMENT
The research was supported by the National Key R&D Program China (Grant No. 2022ZI0010104)), the Science and Technology Commission of Shanghai Municipality (Grant No. 22D21100102), and Shanghai Rising Star Program (Grant No. (QQD140000),
REFERENCES
Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong Qiang Xu. Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pp. 11621-11631, 2020.
Xinyu Cai, Wentao Jiang, Runsheng Xu, Wenquan Zhao, Jiaqi Ma, Si Liu, and Yikang Li. Analyzing infras- tructure lidar placement with realistic lidar simulation library. In 2023 IEFE International Conference on Robotics and Automation (ICRA), pp. 5581-5587, 2023. doi: 10. 10.1109/1RA44481... 220.3.011122
Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Voxel r-enn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1201-1209, 2021.
Kangle Deng, Andrew Liu, Yan Zhu, Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 12882-12891, 2022.
Yinpeng Dong, Caixin Kang, Jinlai Zhang, Zijian Zhu, Yikai Wang, Xiao Yang, Hang Su, Xingxing Wei, and Jun Zhu. Benchmarking robustness of 3d object detection to common corruptions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1022-1032, 2023.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladien Koltun. Carla: An open urban driving simulator. In Conference robot learning, pp. 1-16. PMLR, 2017.
Bo Fu, Zhangjie Cao, Jianmin Wang, and Mingsheng Long. Transferable query selection active domain adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp- 7272-7281, 2021.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3354-3361. IEEE, 2012.
Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow. Digging into self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3828-3838, 2019.
Benjamin Graham. Martin Engelcke, and Laurens Van Der Maaten. 3d semantic segmentation with submani- fold sparse convolutional networks. In Proceedings of the IEFEE conference computer vision and pattern recognition, pp. 9224-9232, 2018.
10

---

Published as a conference paper at ICLR 2024
Michael Gschwandtner, Roland Kwitt, Andreas Uhl, and Wolfgang Pree. Blensor: Blender sensor simulation toolbox. Advances Visual Computing: 7th International Symposiun, ISVC 2011, Las Vegas, NV, USA, September 28, 2011. Proceedings, Part 7, pp. 199-20 208. Springer, 2011.
Jianfei Guo, Nianchen Deng, Xinyang Li, Yeqi Bai, Botian Shi, Chiyu Wang, Chenjing Ding, Dongliang Wang. and Yikang Li. Streetsurf: Extending multi-view implicit surface reconstruction to street views. arXiv preprint arXiv:2306. 04988, 2023.
Yuenan Hou, Xinge Zhu, Yuexin Ma, Chen Change Loy, and Yikang Point-to-vort knowledge distillation for lidar semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8479-8488, 2022.
Siyuan Huang, Zhang. Botian Shi, Hongsheng Li, Yikang Li, and Peng Gao. Sug: Single-dataset unified generalization for 3d point cloud classification. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 8644-8652, 2023.
Xin Li. Yeqi Bai, Pinlong Cai, Licheng Wen, Daocheng Fu, Bo Zhang, Xuemeng Yang. Xinyu Cai. Tao Ma, Jianfci Guo, et al. Towards knowledge-driven autonomous driving. arXiv preprint arXiv:2312. 2,0416 2023.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Lecrning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105. PMIR, 2015.
Sivabalan Manivasagam, Shenlong Wang, Kelvin Wong. Wenyuan Zeng. Mikita Sazanovich, Shuhan Tan, Bin Yang. Wei-Chiu Ma, and Raquel Urtasun. Lidarsim: Realistic lidar simulation by leveraging the real world. 2020 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11173, 2020.
Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chao- qiang Ye, Wei Zhang, Zhenguo Li, et al. One million scenes for autonomous driving: Once dataset. arXiv preprint arXiv:2106. 1037, 2021.
Sajjad Mozaffari, Omar Y Al-Jarrah, Mehrdad Dianati, Paul Jennings. and Alexandros Mouzakitis. Deep learning-based vehicle behavior prediction for autonomous driving applications: review. IEEE Transac- tions Intelligent Transportation Systems, 23(1):33 47, 2020.
Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings the IEFF//VV International Conference on Computer Vision, pp. 5589-5599, 2021.
Jeong Joon Park. Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. (65,174,22 2019.
Konstantinos Rematas, Andrew Liu, Pratul Srinivasan, Jonathan Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12932-12942, 2022.
Santiago Royo and Maria Ballesta- Garcia. An overview of lidar imaging systems for autonomous vehicles. Applied sciences, 9(1)):4993, 2019.
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng Pv-renn: Point-voxel feature set abstraction for 3d object detection. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10529-10538, 2020.
Shaoshuai Shi, Jiang, Jiajun Deng, Zhe Wang. Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn++: Point-voxel feature set abstraction with local vector representation for 3d object detection. arXiv preprint arXiv:2102.0044 2021.
Shaoshuai Shi, Jiang, Dengxin Dai, and Bernt Schicle, Motion transformer with global intention localization and local movement refinement. Advances in Neural Information Processing Systems, 35:6531-6544 2022.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Pamaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2446-2454 2020.
Siyu Teng, Xuemin HU, Peng Deng, Bai Li, Yuchen Li, Yunfeng Ai, Dongsheng Yang. Lingxi Li, Zhe Xu- anyuan, Fenghua Zhu, et al. Motion planning for autonomous driving: The state of the art and future perspectives. IEEE Transactions Intelligent Vehicles, 2023.

---

Published as a conference paper at ICLR 2024
Ignacio Vizzo, Tiziano Guadagnino, Jens Behley, and Cyrill Stachniss. Vdbfusion: Flexible and efficient tsdf integration of range sensor data. Sensors, 22(3):1296, 2022.
Fei Wang. Yan Zhuang, Hong Gu, and Huosheng Hu. Automatic generation of synthetic lidar point clouds for 3-d data analysis. IEEE Transactions on Instrumentation and Measurement 68(7):2671-2677 2019.
Peng Wang, Lingji Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neu- ral implicit surfaces by volume rendering multi-view reconstruction. arXiv preprint arXiv:2106. 10689, 2021.
Yan Wang. Xiangyu Chen, Yurong You, Erran Li, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger, and Wei-Lum Chao. Train in germany, test in the usa: Making object detectors generalize. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11713-11723 2020.
Yi Wei, Zibu Wei, Yongming Rao, Jiaxin Li, Jie Zhou, and Jiwen Lu. Lidar distillation: Bridging the beam- induced domain gap for 3d object detection. arXiv preprint arXiv:2203. 14956, 2022.
Qiangeng Xu, Yin Zhou, Weiyue Wang, Charles R Qi, and Dragomir Anguclov. Spg: Unsupervised domain adaptation for 3d object detection via semantic point generation. In Proceedings of the IEEE/CVF Interna- tional Conference Computer Vision, pp. 15446-15444 2021a.
Runsheng Xu, Yi Guo, Xu Han, Xin Xia. Hao Xiang. and Jiaqi Ma. Openeda: an open cooperative driving automation framework integrated with co-simulation. In 202/ IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 1165-1162. IEEE, 2021b.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10))3337, 2018.
Jihan Yang. Shaoshuai Shi, Zhe Wang. Hongsheng Li, and Xiaojuan Qi. St3d: Self-training for unsupervised domain adaptation on 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10368 10378, 2021.
Jihan Yang, Shaoshuai Shi, Zhe Wang. Hongsheng Li, and Xiaojuan Qi. St3d++: Denoised self-training for unsupervised domain adaptation on 3d object detection. IEEE Transactions on Pattern Analysis Machine Intelligence, (01):1-17, 2022.
Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1389-1399, 2023.
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Pro- ceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11784-11793 2021.
Jiakang Yuan, Bo Zhang, Xiangehao Yan, Tao Chen, Botian Shi, Yikang Li, and Yu Qiao. Ad-pt: Autonomous driving pre-training with large-scale point cloud dataset. arXiv preprint arXiv:2.006.000 006/2, 2023a.
Jiakang Yuan, Bo Zhang, Xiangchao Yan, Tao Chen, Botian Shi, Yikang Li, and Yu Qiao. Bi3d: Bi-domain active learning for cross-domain 3d object detection. In Proceedings of the IEEF/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 15599-15608, 2023b.
Bo Zhang, Jiakang Yuan, Botian Shi, Tao Chen, Yikang Li, and Yu Qiao. Uni3d: A unified baseline for multi- dataset 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9253-9262, 2023.
Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin Oswald, and Mare Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pp. 12786-12796, 2022.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 POINT-TO-MEST DETAILED IMPLEMENTATION
Source Domain Dataset Selection. For selecting the dataset for source domain reconstruction, primary focus is on whether the LiDAR has a sufficiently large vertical Field of View (vFoV), at least equal to or larger than that of the target domain. If the source domain cannot capture informa- tion from a broader area, it will be incapable of acquiring reconstruction information related to the target domain's range. This limitation manifests in the reconstruction process as artifacts or voids appearing in the grid in areas beyond the point cloud FoV of the source domain or in distant regions. To avoid incorporating these parts of the region into the target domain data during the LiDAR ren- dering process, we ensure that the target domain's 's vFoV is less than or equal to that of the source domain's LiDAR.
The parameters of the LiDAR systems in Waymo (Sun et al., 2020), nuScenes (Caesar et al., 2020), and KITTI (Geiger et al., 2012) are as shown Table 5:
Table 5: Vertical Field of View (vFoV) ranges for various datasets.
Dataset Waymo
vFoV
top side x 4
[1,.,6 2.4]
[-90, 30]
KITTI
[-24.9,
nuScenes
1-30.67, 10.67]
SDF Reconstruction. We show some details of point-to-mesh implicit reconstruction. We employ a Signed Distance Field (SDF) to implicitly encode the geometry of street scene objects from Li- DAR point cloud data by training a multi-layer fully-connected neural network to fit the target SDF Specifically, the surface S of the object is represented by the zero-level of its SDF as follows:
S = fx R
SDF(()= = =},
(6)
where the SDF values S; are mapped to a for volume rendering as follows:
4 (S (6)=))))
4
S
.0) .
0₁ max
(7)
+ S
where is a learnable scaling parameter of the Sigmoid function (x) (1 (146-"2)-1 This mapping strategy ensures unbiased calculation color contribution (i.e., visibility weights) while respecting occlusion.
The Computational Requirement for Mesh Reconstruction. For each scene, we train LINR from scratch without any pre-training. Under the LiDAR-only setting, the computational cost required for each sequence's reconstruction task is:
< 2 hrs training time single RTX3090
about 16 GiB GPU Memory
24 GiB CPU Memory (caching data to speed up)
Point Cloud Registration. The Waymo dataset provides precise vehicle and LiDAR poses for every frame in all sequences. It is worth mentioning that, to avoid motion blur, before consolidating all frames, we first remove dynamic objects from the point cloud and only stitch together multiple frames the static background. We further filter out outlier point clouds the scene using point neighborhood statistics. Subsequently, align the point cloud data from each frame in the current sequence to the starting frame of the sequence by projecting them using the LiDAR poses, thus achieving point cloud registration.
Traffic Flow Reconstruction. Due to the strong correlation between vehicle movement and road network structure, vehicles generally cannot travel to non-driving areas. After complete the
13

---

Published as a conference paper at ICLR 2024
Table 6: The sorting results of reconstruction scores (97%) using Root Mean Square Error (RMSE)
and Chamfer Distance (CD).
Sequence Name
RMSE
CD Distance
seg2752216
0.0434
0.0050
seg3451017
0,0590
0.0152
seg1066482
0.0596
0.0111
seg6343780
0.0659
0,0288
seg 125208
0.0672
0.0177
seg 2255991
6,3363
11.7660
seg (48592
7.3421
42,9982
seg 158750
8,4834
127.2535
background reconstruction including roads with the source domain dataset, if the matching of vehi- cle traffic flow cannot be ensured in rendering, it may result in abnormal vehicle movement, such as collisions with surrounding buildings. Therefore, it is crucial to ensure the same density of vehicle traffic flow as source domain dataset. With the information update in the source domain, simulate the traffic flow in CARLA.
each target object the position
Reconstruction Metric. Root Mean Square Error (RMSE) we used for evaluating reconstruction meshes can be formulated as follows Godard et al. (2019):
Σ EN D.)
RMSE
(8)
N
where D: represents the rendered depth, and D; is the ground truth depth obtained from LiDAR.
When computing Chamfer Distance (CD), we consider the upper 97% effective points to deter- mine the average value of the CD distance, while disregarding outliers characterized by significant errors caused by complicated 3D scenes.
A.2 LIDAR RAY CAST PRINCIPLE
Utilizing ray cast tracing to emulate the Time of Flight (ToF) technique (Royo Ballesta-Garcia, 2019), CARLA generates authentic synthetic point clouds by simulating the interaction between rays and object collisions. The LiDAR sensor employs laser emissions scan a specified region, following predetermined angular increment across spherical plane. The orientation of the laser governed by azimuth angle and polar angle é. By extracting the distance r through ray cast operations, the coordinates of point p, be computed as follows:
Tí
cos 02 cos Φ; cos 0, sin d; sin 8
Y1
r1
(9)
24
We utilize the official specifications of the LiDAR used in the target domain to complete the mod- cling of scanning characteristics, while also considering the simulation physical properties. The key technical details of this part fully elaborated in previous work on LiDAR simulation (Cai et al., 2023).
A.3 DATASET DESCRIPTION
Waymo Open Dataset. Waymo Open Dataset (Sun al., 2020), a large-scale autonomous driving dataset collected using 64-beam LiDAR sensor, covers total of 1 1550 scene sequences, which are further divided into train set with 798 sequences, validation set with 202 sequences, and a test set with 150 sequences. Each sequence spans approximately 20 seconds and contains 200 frames of point clouds.
nuScenes Dataset. nuScenes (Caesar al., 2020) collects point clouds using 32-beam LiDAR sensor, consisting of 28130 training samples and 6019 validation samples. Besides, it encompasses 1000 driving scenarios collected in both Boston and Singapore.
14

---

Published as a conference paper at ICLR 2024
KITTI Dataset. KITTI (Geiger al., 2012), collected in Germany, contains point cloud data captured by a 64-beam LiDAR. consists of 7481 training samples and 7581 test samples, where the train set is further divided into 3712 and 3769 samples for training and validation, respectively.
ONCE Dataset. ONCE (Mao et al., 2021) is large-scale autonomous dataset collected China using 40-beam LiDAR sensor. It encompasses diverse range of data collected various times, un- der different weather conditions, and across multiple regions. The dataset covers massive unlabeled point clouds (over 1M frames of point cloud data) and approximately 15K labeled point clouds.
A.4 MORE VISUALIZATION RESULTS
Here, we illustrate more visualization results of the proposed simulation dataset and the reconstruc- tion meshes in Figs. 7 and 8, Also, by visualizing the results, we can observe the shortcomings of the CARLA simulation engine: lacking the authenticity
outdoor scenes. For example, by comparing the CARLA-default and ReSimAD in Fig. 7, it can be seen that the visualization results present the circular-like simulated points using the CARLA-default method. This is mainly due to that current simulator is unable to simulate realistic background information such as road structure.
CARLA-deault
CARLA
Aclaaul delaul
ReSimAD
ReSimAD
nuScenes
(b) KITTI
Figure 7: Visualization of simulated points using CARLA-default background reconstructed background. The 1st and 2nd row indicates the simulated results using CARLA-defauli background and reconstructed background, respectively. The last represents the reconstructed background.
CARLA AARLA-default default
CARLA-defaull
ReSimAD
ReSimAD
(a) nuScenes
(b) KITTI
Figure 8: Visualization of simulated points using CARLA-default background or reconstructed
background. The 1st and 2nd row indicates the simulated results using CARLA-default background
and reconstructed background, respectively. The last row represents the reconstructed background.
15


Reviews:
[{"review_id": "mgePA656mZ", "reviewer": "Reviewer_7ZCS", "summary": "This paper proposes RESIMAD, which aims to base source 3D reconstruction for target-domain point clouds data generation. Specifically, using implicit neural fields with accumulated point clouds per sequence, a high-quality source-domain 3D mesh could be extracted as background of AD scenes. Foreground synthetic assets like vehicles are added within simulation guided by source GT information. Therefore, target-domain point clouds could be further extracted directly or rectified with domain-specific sensor specs. Experiments compared against UDA baseline and pre-training effectiveness demonstrate the potential usage of RESIMAD.", "questions": "Please see the weaknesses section above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The overall paper is well motivated and easy to follow. The RESIMAD pipeline of is composed of three key components and clearly discussed with adequate implementation description.\n\n- The experiments on several Waymo to other datasets setup shows the effectiveness of RESIMAD.", "weaknesses": "1. Effectiveness on larger detection models. One potential benefit of a ‘reconstruction, simulation and perception’ pipeline is that numerous data could be generated/rendered. As only a typical point RCNN baseline model is chosen for evaluation, it would further highlight the advantages of such methods considering more powerful and data-hungry models (e.g., ViT, DETR like detection models). \n\n2. Only ‘zero-shot’ performance is given. It would still be valuable to see whether RESIMAD could benefit from increasingly more target-domain information, from sensor specs (almost zero shot), to few-shot set-up or even with more target samples available)\n\nAbove tow points also applies to the pre-training experiment in Table3.\n\n3. Comparison against more recent DA/UDA methods. UDA is a popular topic attracting much attentions, it is expected to compare against more recent SOTA DA/UDA approaches to further support the evaluation.\n\n4. Although an almost ‘zero-shot’/unsupervised DA could be achieved, such zero-shot performance, I would say, requires the underlying similarity of driving scenarios and a large amount of geometric observations (to form background 3d meshes). I am wondering if the source domain shifts from Waymo to nuScenes or KITTI, will the proposed method still able to work well?\n\n5. Related to last point, one of my main concern is that the acquisition of 3D meshes. It may require a lot of computational costs and multi-lidar sensor specs for pre-training (per-scene optimization of LINR) to get relatively complete and accurate 3d meshs?\n\nEffectiveness and contributions of the proposed methods could be further improved in the above aspects in my opinion. I would like to hear from from authors in the rebuttal phase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tU3ufSpZL4", "reviewer": "Reviewer_kWr7", "summary": "This is a system paper. The authors study a weakened domain generalization setting for 3d object detection from lidar point clouds. For example, training only on Waymo dataset and only accessing the target domain's lidar statistics, the setting pursues good performance on nuScenes. Since information about the target domain is not completely unknown, this is a weakened domain generalization setting. The proposal is to reconstruct the mesh using aggregated LIDAR (or RGB? as NeuS is mentioned). Then the background mesh is put into Carla and car assets are placed according to object size matching. The lidar signal is simulated using the composed scene in Carla. Then the authors train detectors using these simulated data and show the results out-perform the UDA baseline.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "+ The idea generally makes sense and the authors benchmark it in a large-scale, showing meaningful margins.", "weaknesses": "- The biggest issue is a lack of clarify, for both the mesh reconstruction and the sampling part:\n* The reconstruction does not define input, output and losses formally. The only equation is depth rendering. Aggregated lidar piont clouds are used for training? (this is highlighted in bold texts) The authors mention NeuS and I am not sure whether RGB rendering losses are used. \n* Lidar rendering is difficult and there are some sphiscated methods to simulate second-returns [A]. Again the lidar rendering part does not contain any formal mathematical exposition so I cannot understand what the lidar rendering formulation is.\nSince two major algorithmic parts are not understandable, I am rating presentation as poor.\n\n- Let alone the presentation issue about algorithms, I will just assume that these two parts invoke some black-box functions and only consider the system. In this regard, I find the mesh of poor quality. This is understandable as recent papers from my group can only reconstruct meshes from lidar with similar quality. Having that said, I cannot understand why points generated from them are meaningful for detection. The authors should present a systematic evalution for mesh quality (Table.6 does not make sense to me) and rendered point cloud quality. Some comparisons are also needed, e.g., comparing with point clouds rendered from VDBF?\nThis concern makes me rate soundness as fair.\n\nMinor but still confusing issues:\n- Fig.2 is confusing. I cannot understand what the differences are except for the figure color.\n- Table.1 gives literally no additional information since target domains are already mentioned in texts. And Waymo is also target domain？\n\n[A] Neural LiDAR Fields for Novel View Synthesis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mk4s7iHXJ8", "reviewer": "Reviewer_bhEL", "summary": "This paper introduces 3d domain transfer pipeline, ReSimAD.\nIt focuses on bridging the gap between old autonomous driving datasets and newly collected real-life datasets.\nThe pipeline is threefold, 1) point-to-mesh implicit reconstruction, 2) mesh-to-point rendering, and 3) zero-shot perception process.\nFirst two stages are designed to simulate target domain's LiDAR configuration using source domains. For better granulity compared to using raw sparse LiDAR points, it utilizes implicit SDF representation.\nThe last stage is to use the simulated dataset for 3D detection method and perform zero-shot inference on the real dataset.", "questions": "Addressed in weakness section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "This paper suggests dataset generation/simulation pipeline that may enrich/adjust old annotated dataset to the new target domain.", "weaknesses": "1. Poor presentation\nOverall, the placement of figures and tables is not aligned with the text, thereby the whole paper is difficult to follow.\n\n2. Technical novelty & Writing\nThe paper mainly focuses on the dataset simulation process using old annotated dataset.\nTherefore, the most of the methodology is restricted to step-by-step instructions, rather than providing theoretical insights or verifications.\nI believe the paper's contribution on introducing new dataset simulation pipeline does not exceeds its lack of technical novelty.\n\n3. Questionable dataset selection\nSince Waymo dataset is more recent and contains more LiDAR sensors all around the vehicle, compared to nescenes or KITTI, wouldn't it be more plausible to simulate Waymo from KITTI, rather than KITTI from Waymo, to be more in coherence with the paper's motivation?\nIt seems like the pipeline only focuses on interchanging sensor configuration, using the richest point cloud information. Please elaborate.\n\nOn the minor note, I believe that this paper is more related to computer vision or robotics field than machine learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aFu9Q3S2RB", "reviewer": "Reviewer_Fz1j", "summary": "The paper introduces ReSimAD, a unified reconstruction-simulation-perception paradigm that addresses the domain shift issue when 3D detectors face cross-domain performance degradation in the field of autonomous driving.\nThe experimental results conducted on multiple 3D detectors and datasets verify that the proposed paradigm can maintain cross-domain performance in the simulated target domain. In addition, it can also assist and accelerate model training optimization, which has a certain application value.", "questions": "1)\tFigure 1 and 3 can be made more illustrative of the proposed paradigm.\n2)\tThe formatting of the paper needs adjustment, for example, ensure that figures and corresponding sections are not too far apart.\n3)\tTo eliminate artifacts in Point-to-mesh Implicit Reconstruction, it should be further explained how the authors performed point cloud registration when consolidating all frames from the corresponding sequence.\n4)\tThe interpretation of \"Closed Gap\" in Table 2 and its analysis are not provided in the paper.\n5)\tThe paper does not mention the significance or the specific impact of \"the matching of vehicle traffic flow density\" mentioned in Section 4 (Mesh-to-point Rendering).\n6)\tIn Table 5, what is the reason of the significant gap between using zero-shot and oracle methods? The details regarding the sample quantity and other experimental settings can be supplemented in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Regarding the phenomenon of cross-domain performance degradation in 3D detectors in the field of autonomous driving, this paper proposes a paradigm of reconstruction-simulation-perception from the perspective of data sources. This paradigm can alleviate or partially solve the problem of cross-data domain discrepancies. The proposed approach, named ReSimAD, has significant research value.", "weaknesses": "1)\tThe readability of the paper needs improvement, such as the illustrations, explanations, and better formatting.\n2)\tSome necessary elaborations and supporting evidence need to be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)