X-Drive: Cross-modality Consistent Multi-Sensor Data Synthesis for Driving Scenarios
Authors: Yichen Xie
Chenfeng Xu
Chensheng Peng
Shuqi Zhao
Nhat Ho
Alexander T. Pham
Mingyu Ding
Masayoshi Tomizuka
Wei Zhan
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion models; multi-modality data; autonomous driving
Primary area: generative models
TL;DR: We propose a novel framework for the joint generation of point clouds and multi-view images with cross-modality consistency.

Abstract: Recent advancements have exploited diffusion models for the synthesis of either LiDAR point clouds or camera image data in driving scenarios. Despite their success in modeling single-modality data marginal distribution, there is an under- exploration in the mutual reliance between different modalities to describe com- plex driving scenes. To fill in this gap, we propose a novel framework, X-DRIVE, to model the joint distribution of point clouds and multi-view images via a dual- branch latent diffusion model architecture. Considering the distinct geometrical spaces of the two modalities, X-DRIVE conditions the synthesis of each modality on the corresponding local regions from the other modality, ensuring better alignment and realism. To further handle the spatial ambiguity during denoising, we design the cross-modality condition module based on epipolar lines to adaptively learn the cross-modality local correspondence. Besides, X-DRIVE allows for controllable generation through multi-level input conditions, including text, bounding box, image, and point clouds. Extensive results demonstrate the high-fidelity synthetic results of X-DRIVE for both point clouds and multi-view images, adhering to input conditions while ensuring reliable cross-modality consistency. Our code will be made publicly available at https://github.com/yichen928/X-Drive.

Published as a conference paper at ICLR 2025
X-DRIVE: CROSS-MODALITY CONSISTENT MULTI- SENSOR DATA SYNTHESIS FOR DRIVING SCENARIOS
Yichen Xie'r, Chenfeng Xul Chensheng Peng'. P Zhao', Nhat Ho? Alexander T. Pham2, Mingyu Ding't, Masayoshi Tomizuka', Wei Zhan' UC Berkeley
UT Austin Toyota North America
Katdell
abby point ckozd
esly comera Imagy
naky point cledds
Wost
independest
M
(a) Single-modality images and point clouds synthesized (b) Multi-modalie images and point clouds jointly by separate models (Gao et al., 2023; Hu et al., 2024a) generated by our proposed X-DRIVE.
Figure 1: X-DRIVE simultaneously generates high-quality multi-view images and point clouds with cross-modality consistency, which impossible for previous single-modality generative models.
ABSTRACT
Recent advancements have exploited diffusion models for the synthesis of either LiDAR point clouds camera image data in driving scenarios. Despite their success in modeling exploration in the mutual reliance between different modalities describe com- plex driving scenes. To fill in this gap, we propose a novel framework, X-DRIVE, model the joint distribution of point clouds and multi-view images via dual- branch latent diffusion model architecture. Considering the distinct geometrical spaces of the two modalities, X-DRIVE conditions the synthesis of each modality on the corresponding local regions from the other modality, ensuring better align- ment and realism. To further handle the spatial ambiguity during denoising, we design the cross- modality condition module based on epipolar lines to adaptively learn the tross-modalln local correspondence. Besides, X- DRIVE allows con- trollable generation through multi-level input conditions, including text, bounding box, image, and point clouds. Extensive results demonstrate the high-fidelity syn- thetic results of X-DRIVE for both point clouds and multi-vies images, adhering to input conditions while ensuring reliable cross modality consistency. Our code will be made publicly available at http:://gttubb.com//ccce122/////
data marginal distribution, there is an under-
INTRODUCTION
Autonomous driving vehicles perceive the world with multiple sensors of different kinds, where LiDAR and cameras play crucial roles by capturing point clouds and multi-view images. They provide complementary geometric measurements and semantic information about the surrounding environment, significantly benefiting tasks such object detection (Liu et al., 2023c; Xie et al., 2023), motion planning (Sobh et al., 2018), scene reconstruction (Huang et al., 2024; Zhou et al., 2024), and self-supervised representation learning (Yang al., 2024a; Xie al., 2024b). However, these advancements hinge access to large amounts of aligned multi-modality data, specifically well-calibrated LiDAR and multi-view camera inputs that describe the same scene.
Scaling up the collection of such high-quality multi-modality data is costly and non-trivial effort. The high-quality sensors are expensive, and the calibration process demands intensive human ef- forts. Additionally, real-world driving data suffers from the severe long-tail tailed distribution problem, making it an obstacle for corner case collection such as in extreme weather conditions. This raises a natural question: Can we use controllable way synthesize aligned mulli-modality data?

---

Published as a conference paper at ICLR 2025
Given the success in other fields (Rombach et al., 2022; Blattmann et al., 2023; Liu et al., 2023a), generative models offer a promising solution. Current research focuses either on synthesizing point clouds (Zyrianov et al., 2022; Hu et al., 2024a; Ran et al., 2024) multi-view images (Gao et al., 2023; Wang et al., 2023; Wen et al., 2024), with limited attention paid to generating multi-modality data. A simple combination of these single-modality algorithms results in serious cross-modalln mismatches in the synthetic scenes (Fig. la). Such inconsistencies create ambiguities and even contradictory inputs or supervision signals, thus hindering the performance of downstream tasks.
Cross-modality consistency serves as the key desiderata of multi-modality data generation. However, there are several challenges in the generation of consistent LiDAR and camera data. Firstly, synthetic point clouds and multi-view images must be spatially aligned in all the local regions since they describe the same driving scene, iee. the shapes and layouts of both foregrounds and backgrounds must be matched. Secondly, unlike those 2D pixel-level tasks (Zhang et al., 2023), point clouds and multi- view images have distinct geometrical spaces and data formats. Multi-view images are represented by RGB values in camera perspective views, while point clouds are XYZ coordinates in the 3D space. Thirdly, the 3D spatial information is ambiguous during generation for both point clouds and multi-view images without reliable point location or pixel depth in the denoising process.
In this paper, we fill in this gap by proposing X- X-DRIVE, a novel framework for the joint generation of LiDAR and camera data, demonstrated in Fig. lb. We design dual-branch architecture with two latent diffusion models separately dedicated to the synthesis of point clouds and multi-view images, while key cross-modality condition module enhances cross-modality consistency between them. To model the joint distribution and ensure local spatial aligmment, we perform the cross-modality conditions locally with only corresponding regions from the other modality considered. An explicit transform bridges the two different geometrical spaces, converting the cross-modality condition to match each other's 's noisy latent space. To handle the positional ambiguity, we resort to a 3D-aware design based on epipolar lines on range images and multi-view images, allowing the cross- cross-attention module to adaptively determine the mross-modality correspondence without explicit 3D positions. In consequence, X-DRIVE is able to exploit existing single-modality data as conditions to seamlessly generate data in the other modality. Additionally, X-DRIVE enhances controllability by introducing 3D bounding boxes for geometrical layout control and text prompts for attribute control (e.g. weather and lighting), enabling more flexible, fine-grained, and precise control over both modalities.
Extensive experiments demonstrate the great ability of X-DRIVE in generating realistic multi- modality sensor data. It notably outperforms previous specialized single-modality algorithms in the quality of both synthetic point clouds and multi-view images. More importantly, for the first time, it demonstrates reliable cross-modality consistency in synthetic scenes with comprehensive and flexible conditions. Our contributions are summarized follows.
We introduce X-DRIVE, a dual-branch multi-modality latent diffusion framework that, for the first time, enables controllable and reliable synthesis of aligned LiDAR and multi-view camera data.
Our mross-modality epipolar condition module bridges the geometrical gap under spatial ambigu- ity between point clouds and multi-view images, significantly enhancing modality consistency.
Extensive experimental results demonstrate the effectiveness of X-DRIVE, with notable MMD (for point cloud generation) and FID (for multi-view generation) improvements, establishing a new state-of-the-ar for remarkable cross-modally consistency.
2 RELATED WORKS
Conditional generation with diffusion models. Diffusion models (Ho et al., 2020; Dhariwal & Nichol, 2021; Podell et al., 2023; Peebles & Xie, 2023) exhibit remarkable ability in generating diverse images by learning progressive denoising process. They yield -tate-of-the-art results in various tasks such as text-to-image generation (Rombach et al., 2022; Nichol et al., 2021), text-to- video generation (Singer et al., 2022; Yu et al., 2022), and instructional image editing (Brooks et al., 2023; Meng et al., 2022). Beyond text condition, several work emerges with the competence in managing additional forms of control signals. Zhang et al. (2023) integrates spatial conditions to pretrained text-to-inn -oaae diffusion model via efficient finetuning. Li et al. (2023); Zheng et al. (2023) condition the image synthesis the fne-grained geometrical annotations to facilitate downstream tasks like 2D object detection. Liu et al. (2023a); Sargent et al. (2024); Liu et al. (2023b); Yang et al. (2024b) introduce 3D-aware diffusion models for novel view synthesis based on single-view
2

---

Published as a conference paper at ICLR 2025
image inputs, while Shi et al. (2023) explores text-conditioned multi- image generation. Unlike these prior work for the image domain, our framework simultaneously generates point clouds and multi-view images conditioned on text prompts and 3D bounding boxes.
Cross-modality data generation. Beyond image domain, generative models are extended to the synthesis of multi-modality data. Hu et al. (2024b); Shao et al. (2024) leverage video diffusion models to estimate sequential depths. Bai et al. (2024) formulates various vision tasks as next token prediction by extracting diverse visual information as unified visual tokens. The cross-modality synthesis between videos and audios also draws great attention. Efforts are devoted to video-to- audio (Zhu et al., 2022), audio-to-ve -ideo (Chatterjee & Cherian, 2020), bi-directional (Chen et al., 2017: Hao et al., 2018), and joint molti-modality (Ruan et al., 2023) generation. However, in all the cases mentioned above, there exists clear cross-modality alignments, such as pixel-to-pixel spatial correspondence for depth estimation and trame-to-frame temporal alignment between audios and videos. In contrast, point clouds and multi-view images are in different geometrical spaces without clear one-to-one local correlation between noisy multi-modality samples.
Generation of vehiele sensor data. Panoramic driving environment is usually perceived by the multi-view cameras and LiDARs on the ego-vehicle. Alternative to the laborious collection and annotation process, many works resort to the fast-growing generative models for the synthesis of either cameras (Swerdlow et al., 2024; Yang et al., 2023; Gao et al., 2023; Lu et al., 2024; Singh et al., 2024) or LiDAR (Zyrianov et al., 2022; Hu et al., 2024a; Zyrianov et al., 2024) sensor data. For multi- view images (Yang et al., 2023; Wang et al., 2023; Gao al., 2023) videos (Wen et al., 2024; Lu et al., 2023), previous work leverages pretrained image diffusion models (Rombach al., 2022) by incorporating extra inter-view modules to ensure the multi-view consistency. For point clouds, Zyrianov et al. (2022); Ran et al. (2024); Hu et al. (2024a) adapt latent diffusion models for the generation of range images (Fan et al., 2021; Li et al., 2016) since this range-view representation shares similar format with RGB images. However, all above methods concentrate on the single-modality data, either LiDAR or camera sensors. Despite various control signals, there exists no guarantee for the gross-modality consistency between point clouds and multi-view images generated by independent single-mos modality models. To this end, we propose to synthesize consistent multi-modality data in joint manner.
3 PRELIMINARY
Latent Diffusion Models. Diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015) learn to approximate a data distribution p(xo) by iteratively denoising random Gaussian noise for T steps. Typically, diffusion models construct diffused input through forward process, which gradually adds Gaussian noise the data according to a variance schedule BI....,Br.
g(xir Xx)
II
q(x. (xx
= -x((x; vI Bix. Be1).
(1)
A=1
Then, the reverse process learns to recover the original inputs by fitting po px (x,-
(x.).
po (xaer) = p(xr) 1o(x (x = (x f), Eo (xi, f))-
Latent diffusion models (Rombach et al., 2022) perform the diffusion process on the latent space instead of the input space to handle the high dimensional data. Specifically, it maps input x with an encoder into the latent space as = Ex)). The latent code z can reconstructed to the input as & = D(z) through decoder D. The forward and reverse processes of latent diffusion models are similar with the original diffusion models by substituting x with latent z in Eq. and Eq. 2.
Range Image Representation. Compatible with the sampling process of LiDAR sensor, range image R E H....xx is a dense rectangular representation for LiDAR point clouds, where H, and W, are the numbers of rows and columns with one channel representing the range and the other representing intensity. The rows reflect the laser beams while the columns indicate the yaw angles. For each point with Cartesian coordinate (x:, y, 2),, it can be transformed to the spherical coordinates (r, 0, Φ) through the following projection.
x2 + 12 +2, arctan(y, x), arctan(z, 3, /2² x²)
3

---

Published as a conference paper at ICLR 2025
Image Diffusion Model
decedor
cunge lenggr
Coudliiee
Multi-riww Image Diffesion Model
Symhoae Crinendeily Caniinet Dal
Pakayy wall---ee Lmage SASCEE
Spathabe Babbricr tmage
Range Imagee Block
Multi-sie Image Block
CLAF
30 3ecee
Suhppler
Figure 2: Overview of proposed X- X-DRIVE framework. We design architecture to generate multi multi-modality data. Cross modality epipolar condition modules (Fig. 3) are inserted between branches enhance the cross-modality consistency.
diffusion model
where r is the range. the inclination, and the azimuth. Inversely, the 3D point clouds are recovered from range images by = cos()) cos( y = r cos( sin (0), = sé)
The range image is produced by quantizing the 0 and é with factors se and sg. For each point with spherical coordinate (r:, 0; Φi), we have (([0:/sp (0)/s()) (ri, d;). The range ri and intensity 4 are normalized (0, 1).
METHODOLOGY
In this section, we present novel framework, X-DRIVE, for joint generation of point clouds and multi-view images. The overall architecture is depicted in Fig. 2. We extend the vanilla diffusion models to model the joint distribution of multi-modality data in Sec. 4.1. Practically, it is achieved by our proposed framework with dual-branch diffusion models in Sec. 4.2. For the cross cross-modesdt modality consistency, we propose cross-mode moos--to epipolar condition module in Sec. 4.3. Our method can also be tailored for cross-modality conditional generation in a zero-shot manner as in Sec. 4.4.
4.1 JOINT MULTI-MODALL GENERATION
Beyond single-modality diffusion models Sec. 3, we give formulation to our proposed X-DRIVE for multi-moss monlii-modality data generation. It aims to approximate a joint distribution for paired data (ro, Co) that describes a specific driving scene, where ro the range image representation of point clouds and C. {x]]] == denotes multi-view images from V cameras at different perspective views.
Given their similar rectangular latent formats, we set a shared noise schedule $1..... Br for range images and multi-view images. The forward processes of either range image multi-view images only depend on its own modality, equivalent with independent forward processes for each modality.
q(re(rr-1) = N (rr: vI 1,11 ((Cr/C--1) = - ((x)) :o; [VI Bix; (3:I)...
(4)
In contrast, the reverse process exploits models 68.: fg. to predict the noise and 6, added to the range image and multi-view images separately. Given the requirements for gross-modality consis- tency, the noise prediction for each modality should take each other into consideration, so the reverse process for either range image or multi-vien images would depend on both modalities.
pe, (r |ra, C.) = (Tr:-1; (r,, Ci,,), Ee, (r,,C., C),
(5)
po. (C=- [C, = (c (Ct, re, t), Eo. (C,, r),)).. ()).
Intuitively, we can implement two diffusion models as ea, and 6a. separately for range images and multi-view images, equipped with encoders COR and CRC for the other modality as extra conditions.
é co, (Te, ccn(Ce, t): = = (no t)..
(6)
4

---

Published as a conference paper at ICLR 2025
Instead of training separate condition encoders, as strong encoders for each modality. Thus, ca, and 68, are adapted as the condition encoders <g. tC), and cg, (r,, correspondingly in each block. Compared to text-to-image condition, notable characteristic LiDAR-camera mutual condition lies in the local spatial correspondence. For example, car captured by LiDAR should share the consistent shape and location with the same car captured by cameras but has little direct relationship with other parts in the images. Due to their different geometrical space, we should rely on a spatial transformation T(-) to explicitly align the geometrical space for mross-mo correspondence. Concretely. Ter transforms the multi-view image conditions the range image space, while TRE transforms the range image conditions the multi-view image space. As result, the condition encoders CCR; CRC can be formulated as follows.
directly treat diffusion models themselves
ccn(Cr, () Ten(ch, (C), CRc(FE, 1) TRa (co, (1,,
In this case, we rewrite the denoising models in Eq. as follows.
é, f0, (r., Tenlee, t)), t). é c9, (C,, (no
(8)
They can be trained with a joint multi-modality objective function LDM-M.
LDM-M = LDM-R+ LoM-c,
LDM-R = -,gte ||s. - 1. LDM-c Ec Cearellea 1eaaeelle--. -6
(9)
4.2 DUAL-BRANCH JOINT GENERATION FRAMEWORK
Our framework developed following Eq. 8, which consists of two denoising models, i.e., co. (-)
and ca,() (-) respectively tailored for range images (Sec. 4.2.1) and multi-view images (Sec. 4.2.2).
4.2.1 DIFFUSION MODEL. FOR RANGE IMAGES
Architecture. We adapt existing latent diffusion model (LDM) (Rombach et al., 2022) to learning the distribution of range image ro R H, given its similar representation format with RGB images. In the first stage, the range image is compressed into latent feature % E R Xer by downsampling factor fr H images from scratch to learn the latent distribution of z Range image reflects the panoramic view of the ago-vehicle, so its left-most and right-most sides are connected. Considering this constraint, et al., 2022) and RangeLDM (Hu et al., 2024a) to replace all the convolutions in both VAE and LDM with horizontally circular convolutions (Schubert et al., 2019) where the left and right sides of the range image are treated as neighbors.
W,/w,. In the second stage, we train an LDM for range
we follow LiDARGen (Zyrianov
In the first stage, we follow the standard protocol to train the VAE including both encoder & and decoder D, by maximizing the ELBO (Rezende et al., 2014). In addition, we attach an adversarial discriminator (Isola et al., 2017: Rombach et al., 2022) to mitigate the blurriness brought by the reconstruction loss. In the second stage, the LDM conditions the synthesis of range images the cross-modalig information from the multi-view image branch (elaborated in Sec. 4.3) as well as the text prompt and 3D boxes. The model is trained with loss function in Eq. 9.
Range-view bounding box condition. Given a 3D bounding box (b;, b, = Uj (e) e
R represents the 3D coordinates of box corners and c; denotes its semantic category. We project each box corner to the range view using Eq. 3 as b; = {(r, f). 55 These range-view corner coordinates are passed through Fourier embedder. Then, the concatenated position embeddings of eight corners are encoded by MIP as h,,,-- For category label c; we follow (Isola et al., 2017; Gao et al., 2023) to pool the CLIP embedding (Radford et al., 2021) of the category name as hf The box and label embeddings are combined and encoded by another MLP,, as hidden vector h
h (li,, h).))) hi Avg(CLIP(ci)), h MM,, (Mne((CCCca(FFurrr((b)))
(10)
We concatenate box hidden vectors (hE from this range-view box encoder with text prompt hidden vectors, which guide the synthesis the latent diffusion model via a cross-attention module.
4.2.2 DIFFUSION MODEL FOR MULTI-VIEW IMAGES
Architecture. To model the distribution of multi-view images C. (x N we tailor the latent diffusion model by introducing extra bounding box and inter-view conditions apart from the cross-modality signals from range images (details in Sec. 4.3). Following standard
5

---

Published as a conference paper at ICLR 2025
Cumrra-o---DAAE Condition
LIDAR MEAAR-t--Canee Camerr Condition
Sipee
Eapette lis
peoject
Cross Ateetic
nagg Kugg
CrocAAteniin
LIARR
LIDAR
Canera
Figure 3: Cross-modality epipolar condition module. We perform mutual conditions locally between LiDAR and camera modalities based on epipolar lines on multi-view image and range image latents.
LDM (Rombach et al., 2022), each image x is first compressed by pretrained VAE to the latent space as z In the training process, we sample the Gaussian noise for each view independently and the denoising model is trained with the objective function in Eq. 9.
Multi-view bounding box condition. The 3D box condition module similar with Sec. 1, but the box positions are transformed to the perspective view of each camera. For each 3D bounding box (b,,c), we project its 3D corner coordinates b, = {(Tg. Vj, 23)] to corresponding cam- era perspective view as by = {(uj, 1/, d where ujz dj are the corner location in the pixel coordinate and d, is the depth. For each camera view v. its synthesis is only conditioned on the 3D bounding boxes with at least one corner projected into the range of its perspective view image. Then, we formulate this perspective-view box encoder similar with Eq. 10.
(11) For each camera view, the box embeddings th are concatenated with the text prompt hidden vectors, which together guide the image generation through a cross-attention module.
h = ((hE, hp.s); hf. = Avg(CLJP(c:),, h = MI.P
Inter-view condition. It is critical to enhance the consistency across different views in the gen- eration of multi-view images Co = {xi]>=1. so we inject inter-view cross-attention module to condition the generation of each camera view its left and right adjacent views ul and vr. Given the small overlapping between field-of-views of adjacent cameras, we split each image latent 6." into left and right halves. For each view its left half z attends to the right half c,el z of its left neighbor view el, while its right half C.v depends on the left half zarr of its right neighbor ur.
Your zi tanh (a.) Concaturideh ([CrossArttl(e (2,,,, (nn CrossAutm (z. innge 250.
(12)
where a sero-initialization gate (Zhang et al., 2023) for stable optimization. Compared to full cross-attention in MriieDrive (Gao et al., 2023), our split strategy significantly reduces the per-scene GPU memory cost of inter-view condition from 1IGB to 3GB with better multi-view consistency.
4.3 CROSS-MODAL.TT EPIPOLAR CONDITION MODULE
The key to multi-modality data synthesis is to boost the cross-modality consistency, which poten- tially relies on cross-modality conditions. Ideally, there an explicit point-to-pixed correspondence between point clouds and multi-view images through camera projection. However, in the denoising process, we are not aware of either the range values of range images or the depths of the multi-view images. Instead, we propose to warp the local features from range images and multi-view images based on epipolar lines through spatial transforms TRC and TeR (Eq. 7) to provide control signals for the synthesis of the other modality, illustrated in Fig. 3.
Hiinera-to-LDAR condition. For each position (o, 0) on the range image latent z!,, we sample R points along the range axis following the linear- increasing discretization (L.ID) (Reading et al., 2021: Liu et al., 2022) with range value Tki 1, ..,........ R. We transform these points ((0, 0, 0,rx) to their 3D coordinates {I yk xx) -==--
2 = Tx cos(d) (in, = Fk cos(o) cos(f), 24 = ra sin(ø). They can be projected to the camera perspective view with the camera parameters (R,: tv, K.) de
(13)
lu
Uk 1 1 K.
R
8k
zx 1 t.)
k 1,2,..,,
(14)
where (Ug, ux). k 1,2..... R are the pixel coordinates for the R points along the LiDAR ray.
They form the epipolar line the camera image x" corresponding to position (6,0) of the LiDAR

---

Published as a conference paper at ICLR 2025
range image. We bilinearly sample the corresponding local features from the image latent z of view v at location (2k, Uk). If a 3D point (Tk. V/. 2x.) is projected into more than one camera view, would simply adopt the average local features. Extra Fourier embedding of T' is added to each sampled feature as an indicator for the range value along the ray. As a result, the entire spatial transform Ten makes the camera image features spatially aligned to the range image features.
(en (o..) =
(u)) + MLP, (Fourier(xx)))]
(15)
We apply a cross-altention module condition the range image feature (d.00))) at coordinate (ø, () on the corresponding transformed local image features (c on the epipolar line.
Lont (1.00) (.).( (anli((ker) CrossAtta (s.). (or((,""))(0))))
(16)
where Oer is a zero-initiaaln pero-initialization gate for the camera-tor to- condition. The model learns the adaptive local correspondence between range image and multi-view images under range ambiguity.
LiDAR-to-camett condition. Inversely, we condition the synthesis of multi-view images the Li- DAR range images using a similar module. For each position (u, v) on the camera image latent f. from view D points are sampled at different depths following the linear-increasing discretization with depth value de, & 1, ...... D. Each point (u, v, d& in the pixel coordinate transformed to 3D coordinate (I's yk 2k) via the corresponding camera parameters (R,, tv, K.).
[Tk
yk
2k R. K [uk
de
uk dh do] -to) A = 1,2,2 1.2,.... D
(17)
The 3D point coordinate is then projected to the LiDAR range view using Eq. as (Øk, O2 Tk). The D points (0k,0k), 1........ D form the epipolar line on the LiDAR range image corresponding to the pixel coordinate (u, e) of camera view v. The local range image features are extracted at each
coordinate (chx, 0&) from range image latent z through bilinear sampling with additional Fourier embedding for the depth value da, k = 1.2, D attached. We can write the spatial transform TRC aligning the range view features to multi-view image features as follows.
TRC (RC(Z (u.e) = (Bilinear-Sample(4..)
(b)) + MLP, (Fourier(4+))))
(18)
Similar with Camera-to-LiDAye condition, a cross-attention module conditions the local camera im- age feature 0e (u,)) coordinate (u, v) spatially aligned range image features Tnc C.v)
(w,o)
(n,) tanli(()..)) CrossAtta (2 (o))) TRC (2E))
(19)
(a,e)
where are is a zero-initialization gate. The cross-attention module adaptively learns the local re- liance of camera multi-view images LiDAR range images without explicit depth cues.
4.4 CROSS-MODALITY CONDITIONAL GENERATION
Besides multi-modality joint generation. X X-DRIVE work camera-to-LiDAR or LiDAR-to- camera conditional generation model in zero-shot manner without targeted training. Given either one of ground-truth LiDAR range image tp or multi-view images Co. change Eq. separately for camera-to-LiDAR or LiDAR-to-camera generation to synthesize the other modality data.
غ c0, (Te, (éa, (Co, 0)), c0, (C., (éc 0)),
(20)
In this case, the trained diffusion model for the input modality serves strong encoder for the condition to enhance the adherence of generation to the input LiDAR or camera conditions.
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUPS
Dataset. We evaluate our method using nuScenes dataset (Caesar et al., 2020). It provides point clouds from a 32- be-beam LiDAR and multi-view images from 6 cameras. We follow the official setting to employ 700 driving scenes for training and 150 scenes for validation. The generation X- -DIVE conditioned on bounding boxes from 10 semantic classes as well as scene descriptions.

---

Published as a conference paper at ICLR 2025
Evaluation Metrics. For For joint modtiitm data synthesis, we evaluate both fingle-modaiig data realism and cross modalityy consistency. For image modality, we report Fréchet Inception Distance (FID) for image synthesis realism. We also utilize View Consistency Score (VSC) (Swerdlow et al., 2024) to evaluate multi-view consistency, which reflects the keypoints correspondence on overlap- ping regions between adjacent camera views (Fig. 5), and we normalize the VSC of real images to 1.0 for clarity. For point clouds modality, we measure the distribution gap between synthetic and real point clouds with Maximum Mean Discrepancy (MMD) and Jensen-Shannon divergence (JSD) following RangeLDM (Hu et al., 2024a). For cross- modasity consistency, we propose a novel metric called Depth Aligmment Score (DAS). We project the synthetic point clouds to multi- view images as sparse depth values and also estimate the image depth using Depth Anythnnn (Yang et al., 2024d). The mean absolute error between the projected and estimated disparities is used as DAS metric.
Baselines. To our best knowledge, we are the first to work on the joint generation of point clouds and multi-view images. For single-modality generation, we compare with state-of-the-art generation al- gorithms for point clouds, i.e. LiDAR VAE (Caccia et al., 2019), LiDARGen (Zyrianov et al., 2022), RangeLDM (Hu et al., 2024a), and multi-view images, L.e. BEVGen (Swerdlow et al., 2024), BEV- Control (Yang et al., 2023), MriieDrive (Gao et al., 2023), with released code or quantitative results on nuScenes dataset. Furthermore, we combine MagieDrive (Gao et al., 2023) and RangeLDM (Hu al., 2024a) respectively for images and point clouds as a multi-modality baseline.
Training Setup. Our X-DRIVE has a dual-branch architecture. We utilize the pretrained weight to initialize the multi-view image branch with other newly added parameters ran- domly initialized. We follow a three-stage pipeline for the training. In the first stage, the VAE is trained from scratch for range images. In the second stage, we train the latent diffusion model in range image branch without cross-modality conditions using the frozen VAE. In the last stage, the entire multi-modality model trained in an end-to-end manner with only the Stable-Diffusion parts frozen. The first two stages can also be replaced with pretrained point clouds diffusion model. We follow MagicDrive (Gao et al., 2023) and Rangel .DM (Hu et al., 2024a) to synthesize 224 400 multi-view camera images and 32 x 1024 point cloud range images.
Table I:
Quantitative comparison with driving data gener- ation algorithms. For each column, the best value high- lighted by bold, and the second best is denoted by underline.
5.2 QUANTITATIVE RESULTS
We report the quantitative results for both multi moltiit generation as well as LiDAR-to-camera or camera- to-LiDAR cross-modality generation.
(b) Point clouds quality.
(a) Image quality.
Methods
MMD.
JSDJ
LIDAR VAE
10-3
Methods
FID4
VSC1
LIDARGen
x 10-a
0.160
LayoutDiffusion
29.64
RangeLDM
1.9 x 10"
0.054
Joint multi-modality generation. As shown in Tab. 1, compared to spe- cialized single-modality generation methods, X- X-DRIVEL+C, as a multi- modality algorithm, achieves compa- rable or even better quality in both synthetic point clouds and multi-view images. Our FID metric is outperformed by MagicDrive (Gao et al., 2023), since we do not have map condition which offers strong control signals. Unlike simple combination of single-modality methods, we can generate point clouds and images with cross-modality alignment, reflected by our superior DAS metric, thanks to our proposed cross-modality epipolar condition module.
WoVoGen
27.60
X-DRIVEC
10x
0.052
BEVControl
25.54
X- X-DRIVEL+C
122 10-
0.052
BEVGen
24.85
0.439
(c) Cross- modality alignment.
MagicDrive
16.20
0.633
Methods
DAS;
X- &-DRIVEL-IC
16.01
0.721
X- X-DRIVELIC
17.37
0.675
MagieDrive + RangeL.DM
|
2.32
X-DRIVEL4C
1.69
Conditional cross-modality generation. As mentioned in Sec. 4.4, X-DRIVE can also work as a LiDAR-to MiDAR-to-camera co-camera or camera-to-LiDAR conditional generation model. In Tab. 1, for single-mod modality data generation, X-DRIVE can outperform previous baselines for both point clouds and multi-view images, demonstrating the flexibility of our proposed algorithm for cross-modality data synthesis.
Table 2: Object-level control.
Object-level controllable generation. X-DRIVE can ad- here to the 3D bounding box conditions in the gener- ative process. Since previous LiDAR generation meth- ods do not show this ability (Hu et al., 2024a; Ran et al., 2024), only compare our methods with multi-view im- age generation algorithm (Gao et al., 2023). For fair comparison with single-oodlln method, we
Methods
mAP
NDS
Oracle
70.5
72.8
MagicDrive
65.2
69.2
X-DRIVE (ours)
65.4
69.6
employ synthetic multi-view images and real point clouds from nuScenes validation set to run
8

---

Published as a conference paper at ICLR 2025
Chatrn-an LIBAK SynWEE polne oliooolnii
L. MDD- Camera rnd radeaso
Figure 4: Qualitative results for multi-modality and conditional cross-modality generation. Colors
of projected point clouds refer to depths. Well-matched regions are highlighted with red circles.
Magktrtte
X-tmee (om))
Figure 5: Key points correspondence between adjacent synthetic camera images. Our proposed
method (bottom) can bring higher multi-view consistency than the baseline (top).
pretrained SparseFusion (Xie et al., 2023) multi-sensor object detection algorithm. Tab. shows
X-DRIVE outperforms MagicDrive (Gao al., 2023) w.r.f the object-level fidelity.
Table 3: Object detection training.
Training support for downstream task. X-DRIVE can produce synthetic multi-modality data to enhance the training for downstream perception models. We train a multi-sensor 3D object detection model using SparseFu- sion method (Xie et al., 2023) with combination of 1.4k
Training data
mAP
NDS
real data only
56.28
51.54
real synthetic data
57.60
52.23
real scenes from nuScenes dataset (Caesar al., 2020) and 1.4k synthetic scenes. Tab. 3 shows that the model trained with mixed data can notably outperform the model trained only with real data. This reflects both single-modaling quality and cross moddalty alignment of our synthetic data.
5.3 QUALITATIVE ANALYSTS
Cross-modality consistency. We show some multi-modality generation examples to exhibit the cross-modality consistency qualitatively, As in Fig. 4, the projected point clouds overlap with multi- view image contents properly for both foregrounds and backgrounds. We also show the results of conditional LiDARR and LiDAR-to-camen generation Fig. 4, where the synthetic data adheres to the cross- modallty LiDAR or camera conditions well.
Multi-view consistency, our multi-modaiee generation, point clouds provide natural 3D- awwre geometrical guidance for the multi-view images, benefiting the multi-view consistency. In Fig. 5, we visualize the keypoints matching results (Sun et al., 2021) between adjacent synthetic camera images, where better correspondence is witnessed for X-DRIVE than MagicDrive (Gao al., 2023).
Scene-level and object-level control. Given the text and bounding box conditions, X-DRIVE gen- erates diverse results using different control signals, illustrated in Fig. 6. For scene-level control, we can target for various lighting and weathers through the text prompts. Synthetic images still remain realistic and adhere to the object layouts, and synthetic point clouds become noisy in rainy weather due to the water in the air or on the ground. For object-level control, we can edit the scene by deleting objects or inserting objects at desired locations with specific sizes and semantic categories.

---

Published as a conference paper at ICLR 2025
10 bax
De Semer agesr
Figure 6: Scene-level and object-level controllable generation by changing the input conditions.
Table 5: Ablations of cross- moos-mm module.
Table 4: Ablations of conditions and training.
Methods
JSD]
Methods
PIDL
JSDJ
we test prompt condition
whf two-stagu training
weo 3D box condition
23.65
29.60
25.76
0.070
0.113
0.091
1.84
1.76
1.73
half sampling points
nof spipolar line
cross-modality
opipolar condition
25.35
20.70
0.088
0.074
2.23
2.25
1.95
full
full
0.070
5.4 ABLATION STUDIES
Due to limited resources, we employ shorter training schedule for ablation studies.
Two-stage training. X-DRIVE benefits from the two-stage training pipeline that trains the LiDAR range image branch and cross-modality modules in sequential manner. In Tab. 4, there is notable performance drop in point clouds quality if we merge the two training stages, since the two-stage training can make the optimization more simple and stable for the LiDAR range image branch.
Bounding box and text prompt input conditions. Although X- E-DRIVE also supports unconditional generation, these input conditions provide multi-level control signals to generate diverse outputs. Tab. shows that text and box conditions improve the quality of synthetic point clouds and images.
Cross-modality condition module. The cross-modality condition module is critical for cross- modality consistency. shown in Tab. 5, removing it significantly hurts the cross-modality align- ment although the shared bounding boxes and text prompts can provide some constraints.
Epipolar-line cross-attention. Tab. justifies our design epipolar-ling cross-attention module to solve the spatial ambiguity (Eq. 16 and Eq. 19). Alternatively, randomly sampling same number of local features from the other modality feature map without epipolar lines cannot ensure the cross- modality alignment. If we reduce the number of sampling points along the epipolar line, the per- formance would also be severely hurt. Besides, the cross-attention is also important for adaptively learning the local correspondence. Simple averaging along epipolar lines instead of cross -attenttton quite noisy and poses severe damage to the point clouds quality and cross-modality consistency.
CONCLUSIONS
We present X- X-DRIVE, novel framework for the multi multi-modality generation of aligned LiDAR point clouds and multi-view camera images. By leveraging a dual-branch diffusion model, X-DRIVE captures the joint distribution of multi-modality data, ensuring mutual dependence between modal- ities. key feature is the cross-modality condition module, which uses epipolar lines on LiDAR range images and camera views to address spatial ambiguity during the denoising process. Addi- tionally, X-DRIVE supports multi-level control, enabling synthesis based on text prompts and object bounding boxes. Extensive experiments show that X- X-DRIVE generates high-quality, geometrically and semantically aligned point clouds and images, maintaining fidelity to the input conditions while accurately describing the same driving scene together.
10

---

Published as a conference paper at ICLR 2025
Acknowledegements 2235013.
This work supported in part by Berkeley DeepDrive and NSF Grant.
REFERENCES
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22861-22872, 2024.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis, Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25563-2257,, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Pruceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Lucas Caccia, Herke Van Hoof, Aaron Courville, and Joelle Pineau. Deep generative modeling of lidar data. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5034-5040. IEBE, 2019.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom, nuscenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1621-11631,, 2020.
Moitreya Chatterjee and Anoop Cherian. Sound2sight: Generating visual dynamics from sound and context. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16, pp. 701-719. Springer, 2020.
Lele Chen, Sudhanshu Srivastava, Zhiyao Duan, and Chenliang Xu. Deep cross-modal audio-visual generation. In Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pp. 349-357, 2017.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 44:8780-8794, 2021.
Alexey Dosovitskiy. An image is worth 16x16 16 words: Transformers for image recognition at scale. arXiv preprint orXiv:2010.1 1929, 2020.
Lue Fan, Xuan Xiong, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Rangedet: defense of range view for lidar-based 3d object detection. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 1918-2927, 2021.
Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit- Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. arXiv preprint arXiv:2310.02601, 2023.
Wangli Hao, Zhaoxiang Zhang, and He Guan. Cmegan: A uniform framework for cross-modal visual-audio mutual generation. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851 2020.
Qianjiang Hu, Zhimin Zhang, and Wei Hu. Rangeldm: Fast realistic lidar point cloud generation.
arXiv preprint ar arX0::2403.1009, 2024a.
Wenbo Hu, Xiangjun Gao, Xiaoyu Li. Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open- open-world world videos. arXiv preprint arXiv:2409.02090 2024h.
11

---

Published as a conference paper at ICLR 2025
Nan Huang, Xiaobao Wei, Wenzhao Zheng, Pengju An, Ming Lu, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. S gaussssn:: Self-super street gaussians for au- tonomous driving, 2024. URL sutttss//aatee /wtpivaaivvv....///a./...../..v.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. maagg-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1125-1134, 2017.
Bo Li, Tianlei Zhang, and Tian Xia. Vehicle detection from 3d lidar using fully convolutional network. arXiv preprint arXiv:160..07916, 2016.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to- eext-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511 22511-22555 1-2221,,,,, 2023.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9298-9309, 2023a.
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In European Conference on Computer Vision, pp- 531-548. Springer, 2022.
Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang- Syncdreamer: Generating multiview-consister images from a single-view image. arXiv preprint 2rXiv:2309.03453, 2023b.
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela I. Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. In 2023 IEEE international conference on robotics and automation (ICRA), pp. 2774-2781. IEEE, 2023c.
Jiachen Lu, Ze Huang. Jiahui Zhang, Zeyu Yang, and Li Zhang. Wovogen: World volume-aware dif- fusion for controllable multi-camera driving scene generation. arXiv preprint arXiv:3312.09934, 2023.
Jiachen Lu, Ze Huang, Zeyu Yang, Jiahui Zhang, and Zhang. Wovogen: World volume-aware diffuuson for controllable multi-camera driving scene generation. In European Conference Computer Vision, pp. 329-345. Springer, 2024.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorcalistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 /0.44 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 7748-8763. PMIR, 2021.
Haoxi Ran, Vitor Guizilini, and Yue Wang. Towards realistic scene generation with lidar diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 11738-14748, 2024.
12

---

Published as a conference paper at ICLR 2025
Cody Reading, Ali Harakeh, Julia Chae, and Steven L Waslander. Categorical depth distribution network for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10219-10228, 2023.
Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, al. Zeronvs: Zero-shot 360-degree view synthesis from single image. In Proceedings of the IEEESCVF Conference on Computer Vision and Pattern Recognition, pp. 4420-9429, 2024.
Stefan Schubert, Peer Neubert, Johannes Pöschmann, and Peter Protzel. Circular convolutional neu- ral networks for panoramic images and laser data. In 2019 IEEE intelligent vehicles symposium (IV), pp. 653-660. IEEE, 2019.
Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Matteo Poggi, and Yiyi Liao. Learning temporally consistent video depth from video diffusion priors. arXiv preprint arXiv:2406.01493, 2024.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. In The Eleventh International Conference on Learning Representations, 2022.
Bharat Singh, Viveka Kulharia, Luyu Yang, Avinash Ravichandran, Ambrish Tyagi, and Ashish Shrivastava. Genmm: Geometrically and temporally consistent multimodal data generation for video and lidar. arXiv preprint arXiv:2406. 10722, 2024.
Ibrahim Sobh, Loay Amin, Sherif Abdelkarim, Khaled Elmadawy, Mahmoud Saced, Omar Ab- deltawab, Mostafa Gamal, and Ahmad El Sallab. End-to-end multi-modal sensors fusion system for urban automated driving. 32nd Conference on Neural Information Processing Systems, 2018.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265. PMLR, 2015.
Jiaming Sun, Zehong Shen, Yuang Wang. Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 9922-8931, 2021.
Alexander Swerdlow, Runsheng Xu, and Bolei Zhou. Street-view image generation from bird's- eye view layout. IEEE Robotics and Automation Letters, 2024.
Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu. Drivedreamer: Towards real-world-driven world models for autonomous driving, arXiv preprint arXiv:2309.09777. 2023.
Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6902-6912, 2024.
13

---

Published as a conference paper at ICLR 2025
Enze Xie. Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffu- sion transformers. arXiv preprint arXiv:2410. 10629, 2024a.
Yichen Xie, Chenfeng Xu, Marie-Julie Rakotosaona, Patrick Rim, Federico Tombari, Kurt Keutzer, Masayoshi Tomizuka, and Wei Zhan. Sparsefusion: Fusing multi-modal sparse representations for multi-sensor 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17591-17602. 2023.
Yichen Xie, Hongge Chen, Gregory P Meyer, Yong Jae Lee, Eric M Wolff, Masayoshi Tomizuka, Wei Zhan, Yuning Chai, and Xin Huang. Cohere3d: Exploiting temporal coherence for unsupervised representation learning of vision-based autonomous driving. arXiv preprint arXiv:2402.15583, 2024b.
Honghui Yang, Sha Zhang, Di Huang, Xiaoyang Wu, Haoyi Zhu, Tong He, Shixiang Tang, Heng- shuang Zhao, Qibo Qiu, Binbin Lin, al. Unipad: A universal pre-training paradigm for au- tonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15238-15250, 2024a.
Jiayu Yang, Ziang Cheng, Yunfei Duan, Pan Ji, and Hongdong Li. Consistnet: Enforcing 3d consis- tency for vue images diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7079-7088, 2024b.
Kairui Yang, Enhui Ma, Jibin Peng, Qing Guo, Di Lin, and Kaicheng Yu. Bevcontrol: Accurately controlling street-view elements with multi-perspective consistency via bev sketch layout. arXiv preprint arXiv:2308.0166// 2023.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 10371-10381, 2024c.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. MXXv:24006009144 2024d.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In International Conference on Learning Representations, 2022.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings the pp. 3836-3847, 2023.
the IEEE/CVE International Conference on Computer Vision,
Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12490-22499 2023.
Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming Hsuan Yang. Driv- inggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 11634-21644 2024.
Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan. Discrete contrastive diffuuion for cross cross-modal music and image generation. In The Eleventh International Conference on Learning Representations, 2022.
Vlas Zyrianov, Xiyue Zhu, and Shenlong Wang. Learning to generate realistic lidar point clouds. In European Conference on Computer Vision, pp. 17-35. Springer, 2022.
Vlas Zyrianov, Henry Che, Zhijian Liu, and Shenlong Wang. Lidardm: Generative lidar simulation in generated world, arXiv preprint arXiv:2404.0 02903, 2024.
14

---

Published as a conference paper at ICLR 2025
In this appendix, first give an extra ablation study about the classifier-free guidance in Ap- pendix A. Then, we elaborate our implementation details in Appendix B. Afterwards, we discuss the limitation of our framework and future works in Appendix C. Finally, more qualitative results of multi-modality data generation are demonstrated in Appendix D.
A ABLATION STUDY OF CLASSIFIER-FREI GUIDANCE SCALES
We employ classifier-free free guidance (CFG) during inference for box and text conditions. Illustrated in Fig. 7, improves the realism of synthetic data. However, higher CFG scale would hurt the image and point cloud quality by increasing the contrast and sharpness.
B IMPLEMENTATION DETAILS
In this part, we explain details in the implementation of X-DRIVE including network architecture, training sched- ule, and evaluation metrics. Our code will be made pub- licly available upon the acceptance of this paper.
6:80
Network architecture. Given the small height dimen- sion of LiDAR range images and limited data amount, we employ smaller sizes for the VAE and latent diffusion model (L.M)) for LiDAR range images. Specifically, the VAE includes three encoder blocks and three decoder blocks with channels [32, 64, 128] separately. There are downsampling modules in the first two encoder blocks. The latent zz for range image has size of 256 8. The range image LDM is composed of three downsampling blocks, block, and three upsampling blocks with attention modules in all the blocks. The channel numbers for three downsampling and upsampling blocks are [128, 256, 512] respectively. The cross- cross-modass modality epipolar condition modules are inserted between each pair of corresponding blocks in point cloud and multi-view image diffusion model branches. For these seven cross-modality modules, we set the sampling number R and D along range and depth axes as [24, 24, 12, 6, 12, 24, 24] separately with both maximal depth and range as 60m.
Qleefe-ttu Scri
Figure 7: Effect of CFG scales.
middle
Training schedule. The training of X-DRIVE on nuScenes dataset (Caesar et al., 2020) includes three stages. Since there are no publicly available pretrained weights for LiDAR diffusion models, we train the VAE and LDM for LiDAR range image in the first two stages separately. Afterwards, we train the entire multi-modality generation framework together. Before training. we initialize our multi-view image diffusion model branch with pretrained the other modules except the zero-ine acro-initializatton gates are initialized randomly. all the stages, our model is trained using NVIDIA RTX A6000 GPUs.
V1.5 weights, while all
In the first stage, VAE for LiDAR range image is trained using batch size 96 and learning rate 4e-4 for 200 epochs. The discriminator takes effect after 1000 iterations. In the second stage, train the LiDAR LDM from scratch using batch size 96 and learning rate le-4 for 2000 epochs. The model includes the text prompt and 3D range-view bounding box condition modules with drop-rate 0.25 for either condition during training. In the last stage, we train the entire framework in an end-to- end manner with module trainable states illustrated in Fig. 2 using condition drop-rate 0,25 as well. The entire model is trained for 250 epochs with learning rate 8e-5 and batch size 24 in our main experiments. For ablation studies, we reduce the epoch number to 80 for efficiency.
Evaluation metries. In our quantitative evaluation, we apply the FID metric from Magic- Drive (Gao et al., 2023), VSC metric from BEVGen (Swerdlow et al., 2024), MMD and JSD met- rics from LiDARGen (Zyrianov et al., 2022). For our proposed DAS metric, we run pretrained DepthAnything model (Yang et al., 2024d) with ViT-B backbone (Dosovitskiy, 2020) on the syn- thetic images to estimate the disparity (Yang et al., 2024c). Then, we project the synthetic point clouds to multi-vien view images to get sparse disparity. We normalize the projected sparse disparity to the same scale as the estimated disparity. The mean absolute error is fetched as our DAS metric.
15

---

Published as a conference paper at ICLR 2025
C LIMITATION AND FUTURE WORKS
This paper concentrates on the joint generation of consistent multi- modiiit data. In this case, due the limitation of accessible computational resources, we do not include some natural extensions of our X-DRIVE framework. Firstly, our method is limited to the generation of single-frame point clouds and multi-view images. In fact, is intuitive combine existing temporal attention modules like Wen et al. (2024) with our X-DRIVE. Secondly, we can also incorporate some extra conditions for data synthesis such as HD maps (Gao et al., 2023) into our framework for more controllable generation. Lastly, due to the distribution of nuScenes training set (Caesar et al., 2020), we cannot synthesize some specific scenarios such as snowy weathers. We will try extend our framework to deal with above limitations.
Our proposed X-DRIVE is orthogonal other efforts improving single-modality generative model performance such as transformer-based image generation framework (Peebles & Xie, 2023), high image resolutions (Xie et al., 2024a), or point clouds representation for LiDAR sensor data. We will combine the cross- mrods-mod condition module of X- X-DRIVE with these advanced techniques to further enhance its performance multi-modality data generation.
D QUALITATIVE RESULTS
In this section, we demonstrate additional examples of our synthetic multi-modality data in Fig. 8. X-DRIVE can generate realistic point clouds and multi-view images with cross-modality consistency based on the input conditions.
16

---

Published as a conference paper at ICLR 2025
Input conditions
Synthetic multi-view images
Synthetic point clouds
ditsing siese insige boston report. flgin, purkid cars, turn lift, pods, fius truck, ped crussing crosewatie, auitriss, comes.
driving imsgee faston-respoit Ptrrled vans, suttell lans, consrrucioon the rights bilg foliowing passal by active monomyylle
driving image foston seaport, Strked stop light, jayuuller, peds, fuz.
A sirving scene image at boston repport Raiin yub, comes parkel cars comstructonn rite, construction volicie, parking (it, barrien,,
driwvng baston Saston-seaport seaport. Change stop, dight traffic congestion due traffic light, pad oserpass, parked carz. eyclist.
A driving scone image faston-noqun, Maix turt right, parfeed tracly.
ola
Figure Additional multi-modality data synthesis results using our proposed X-DRIVE.
17


Reviews:
[{"review_id": "79oRGaS7fJ", "reviewer": "Reviewer_wkMY", "summary": "This paper introduces X-DRIVE, a novel framework for generating consistent multimodal data, specifically LiDAR point clouds and multi-view images, for autonomous driving scenarios. The key innovation of this paper is a cross-modality epipoplar condition module that bridges the geometrical gap under spatial ambiguity between point clouds and multi-view images, significantly enhancing modality consistency. Experimental results have demonstrated the effectiveness of the proposed method.", "questions": "Is there a small mistake in Eq. (5)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper introduces the first framework for the joint generation of LiDAR and camera data for autonomous driving scenarios. \n\n+ The proposed cross-modality epipolar condition module to enhance modality consistency makes sense and works well. \n\n+ The paper is well-written and easy to follow.", "weaknesses": "1. This paper presents a well-designed diffusion architecture for joint LiDAR and image data generation. My major concern is the novelty. This paper's main contribution is a cross-modality epipolar condition module for consistency constraints. However, epipolar attention for multi-view consistency has been extensively explored in multi-view diffusion models [1, 2, 3]. \n\n[1] Liu, Yuan, et al. \"Syncdreamer: Generating multiview-consistent images from a single-view image.\" arXiv preprint arXiv:2309.03453 (2023).\n[2] Shi, Yichun, et al. \"Mvdream: Multi-view diffusion for 3d generation.\" arXiv preprint arXiv:2308.16512 (2023).\n[3] Yang, Jiayu, et al. \"Consistnet: Enforcing 3d consistency for multi-view images diffusion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n2. Experiments are only conducted on one dataset, nuScenes. \n\n3. It would be great to detail what kind of and how many GPUs are used, as well as the training/inference time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yMPDqJXQLo", "reviewer": "Reviewer_vwWu", "summary": "- This work proposes a novel technique to perform driving based lidar data generation in the form of range images, and multi-camera data generation while ensuring cross-modal alignment by leveraging latent diffusion models.\n- Prior works address multi-camera image generation and lidar based generation individually while this work aims to perform joint generation of lidar and camera data. The major contribution here is in aligning the modalities during generation.\n- During denoising of a first modality (camera or lidar), the information from the other modality (lidar or camera) is encoded and transformed into the geometrical space of the first modality (camera or lidar) to ensure alignment. They do this by finding matches from a second modality for a query from first modality by traversing the epipolar line in the second modality corresponding to the query.\n- They show qualitative and quantitative results for conditional and unconditional generations. In addition, the authors also display controllability via bounding boxes and language interfaces.", "questions": "- Q1: Baseline setting for Tab 1: I don’t completely understand how MagicDrive + RangeLDM was combined to obtain cross-modality score? Was there anything done to ensure alignment between the different modalities from the two models?\n- Q2:  I was wondering if the authors made the decision of studying this problem in range-image space based on empirical results or out of convenience (prior work)? In other words, was denoising in pointcloud space studied for this problem?\n- Q3: (suggestion) I wonder if the evaluation metrics are unable to capture objectness of pointclouds. Athough not typical in generative learning, techniques like Iterative closest point (ICP) could be useful in measuring shape discrepancies.\n- Q4: Performing denoising in each modality while encoding information from the other modality through the denoising time steps seems expensive. It would be helpful to provide the computation time split between the different stages? If possible, also relate it to time taken for unimodal generations.\n- Q5: Was the zero initialization from ControlNet deemed necessary empirically? I’m curious if the authors observed any forgetting without it.\n- Q6: It looks like the results for lidar to camera is far better than the multimodal and camera to lidar results to me. Do the authors agree? If so, since generating point clouds is difficult, would disnetangling the joint process to a sequential approach of first generating camera data and then performing conditional lidar generation be feasible?\n- Q7: Have the authors looked at GenMM [6]? While they do not specifically tackle multi-modal generations, they target alignment between the two modalities using in-painting diffusion models. I wonder on similar lines to Q5 if their approach can be conducive to a sequential approach of using reference patches from camera generations to condition generating point cloud data leveraging in-painting.\n\nPrioritization order: Q1 > Q2 = Q4 > Q7 = Q6 > Q5 > Q3\n\nI welcome the authors to correct me if there was any misunderstanding on my part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- S1: The work addresses an important problem of lacking aligned lidar camera data. Solutions towards this can reduce/elimintate post-processing work of performing motion compensation via scene flow between consecutive pointcloud frames to address the lack of alignment. This is a cumbersome process.\n- S2: The epipolar line based matching technique is neat, intuitive, and has been successfully been used before in literature [1].\n- S3: The manuscript is well written and easy to follow. I also like the multi-view consistency results.\n\n[1] Consistent View Synthesis with Pose-Guided Diffusion Models - https://arxiv.org/pdf/2303.17598", "weaknesses": "- W1: Mapping pointclouds to range images leads to information loss. It is understandable the authors use range images since most of the existing lidar generation work uses range images and in addition, image based generation techniques are much more mature than pointcloud counterparts. However, the major goal here is to ensure alignment between the modalities and not to perform unimodal generations which have been addressed before. Therefore, I would expect point clouds to be effective in filling gaps caused by information loss especially during occlusions and in far ranges.\n- W2: I find the qualitative results of specifically the lidar generations lacking when viewed in the form of point clouds. Since the application of this work is to be used for training models, having distributional similarities (elaborated below) with real points is important to serve as valuable indictive biases and I doubt its utility without it.\n    - Pattern realism: The lidar patterns shown in the pointcloud figures do not look realistic to me. For instance, the ground points on a rotating lidar typically show-up as thick concentric circles whereas results in supp. Fig 8 look very artificial to me. Comparing this to real pointclouds or even other synthetic lidar generation techniques from literature such as [2, 3], I see a gap that might be important to ignore for a downstream model.\n    - Object realism: The point clouds also lack object shape or object realism in the figures presented in Fig 6 of main paper. For instance, vehicles have a typical shape based on their orientation which are useful biases for a downstream network that consumes this data. These patterns are very evident in other synthetic techniques [2, 3] but seems to be lacking here. Similarly, on the range images for the multi-modal generations in Fig 4, very prominent cars in the scene do not have as many lidar points as I would expect. This was an objective of the work as stated in the manuscript (L64) which I don’t believe has been fully fulfiled.\n    - Following similar theme, the points on the pedestrian in Fig 4 do not seem realistic to me.\n- W3: Related to W2, since the larger goal of this work is to generate data that can be used for training, I would have expected to see results showcasing this utility. The manuscript in the current form does not convey the effectiveness of the data towards this.\n- W4: The authors state one of the application is to address long-tail scenarios (L52). They do show results in rain and bounding box can be a way to inject situtations such as cut-ins. However, these are mainly shown in the visual space currently which is also illustrated by prior works such as [4, 5]. The primary evaluation of the work proposed here is not in judging effectiveness of visual generation but alignment between cross-modal generations. The current results in the manuscript do not show such OOD scenarios which can be validated in the point cloud and image space.\n\nI am weighting W2 > W1 > W3 = W4.\n\nOverall, I like the work and the problem being tackled. However, I do not believe the results are quite there yet to be useful. Therefore, I am inclined to reject this work but I am open to changing my opinions based on the rebuttal and reviewer discussion.\n\nMinor comments:\n\n- Typo in L259 - Fourier\n- L294 - I think you mean split vertically?\n- Typo in L424.\n\n\nReferences:\n\n[2] Towards Realistic Scene Generation with LiDAR Diffusion Models - https://arxiv.org/pdf/2404.00815\n\n[3] RangeLDM - https://arxiv.org/pdf/2403.10094\n\n[4] MagicDrive - https://arxiv.org/pdf/2310.02601\n\n[5] BEVGen - https://arxiv.org/pdf/2301.04634\n\n[6] GenMM - https://arxiv.org/pdf/2406.10722", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OxYoYq1OuG", "reviewer": "Reviewer_Tmsf", "summary": "This work proposes X-Drive, explicitly addressing LiDAR-image consistency in driving data synthesis. It designs (1) mutual conditioning between corresponding local regions across modalities, (2) an epipolar line-based conditioning module, and (3) multi-source input conditions to mitigate generation inconsistencies. Experiments on nuScenes validate the efficacy of the proposed method.", "questions": "- In Figure 1(b), what does the distribution at the bottom represent?\n- In Figure 2, how is the generation of the LiDAR point cloud achieved?\n- What are the additional time and memory costs of the designed module?\n- What is the average generation time for one experiment?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The focus on consistency between LiDAR and image generation provides insights to this field.", "weaknesses": "- Lack of quantitative comparison with baselines. In Table 1, the number of experiments conducted is insufficient to fully support this paper. The authors are encouraged to compare with more previous methods and implement additional vanilla baselines to better illustrate the advantages of X-Drive. Table 1 could be divided by modality and enriched separately.\n- Lack of ablation studies. In Table 3, the ablation analysis is presented at a very coarse level. The authors are encouraged to detail the design of each module and analyze their individual impact on the final results.\n- Marginal improvements. In Table 1, for JSD in the C→L setting, X-Drive only surpasses MagicDrive by 0.002, while RangeLDM surpasses LiDARGen by 0.106. For FID in the L→C setting, X-Drive only surpasses MagicDrive by 0.19, whereas MagicDrive surpasses BEVGen by 8.65.\n- Numerous typos. For example, \"several works emerges\" in L103, \"We extend the vanilla diffusion models to modeling\" in L186-187, and \"Our FID metric is outoerformed by MagicDrive\" in L423-424. These errors should be reviewed and corrected to enhance the clarity and readability of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bk3oggQhDt", "reviewer": "Reviewer_SjU7", "summary": "# Review Analysis of X-DRIVE Paper\n\n## 1. Methodology and Key Innovations\n\n### Primary Methods\n- Proposed X-DRIVE framework: A dual-branch latent diffusion model architecture for simultaneous generation of LiDAR point clouds and multi-view camera images\n- Designed Cross-modality Epipolar Condition Module to enhance consistency between different modalities\n- Introduced 3D bounding boxes for geometric layout control and text prompts for attribute control (e.g. weather and lighting)\n\n### Key Innovations\n- First work to achieve joint generation of LiDAR point clouds and multi-view camera images with cross-modal consistency\n- Novel epipolar-based cross-modal condition module addressing alignment between point clouds and images in different geometric spaces\n- Multi-level controllable generation incorporating text, bounding boxes, images and point clouds as conditions\n\n## 2. Innovation Analysis\n\n### Limitations in Innovation\n- Weak theoretical foundation for cross-modal consistency, lacking rigorous proof of epipolar-based design effectiveness\n- Incomplete solution for \"spatial ambiguity\", relying solely on epipolar geometry may not fully resolve 3D space ambiguities\n- Limited innovation in generation control, mainly applying existing control methods to new scenarios\n\n## 3. Experimental Analysis\n\n### Required Additional Experiments\n- Evaluation of generated data utility in downstream tasks (3D object detection, trajectory prediction)\n- Need to adopt more advanced transformer architectures like DiT instead of U-Net\n- Consider replacing CLIP with T5 for potentially better performance\n- Address low resolution issues affecting geometric shapes of distant objects (The current resolution is too low. Objects in the distance have no geometric shapes at all. However, the point cloud may have relatively complete and accurate geometric shapes in the distance. A larger resolution should be used, and because the generated image has slightly less geometric shapes in the distance, the effect is not good, simple and practical super-resolution model. It can only increase the resolution but cannot improve the distant geometry.)\n- Investigate video generation capabilities with 3D VAE and 3D attention mechanisms\n- Develop quantitative metrics for cross-modal consistency evaluation\n\n## 4. Ablation Study Analysis\n\n### Missing Ablation Studies\n- Detailed component-wise analysis of cross-modal epipolar condition module\n- Impact assessment of different condition types on generation quality\n- Sensitivity analysis of key hyperparameters\n- Investigation of different training strategies\n\n## 5. Advanced Experiments (Optional)\n\n### Long-tail Data Generation\n- Potential for generating rare/corner cases\n- Ability to synthesize challenging scenarios\n- Evaluation of model performance on long-tail distributions\n\n## Overall Assessment\nWhile the paper presents novel contributions in cross-modal data generation, it requires strengthening in theoretical foundations, experimental validation, and analysis depth. The framework shows promise but needs additional experiments and rigorous evaluation to fully demonstrate its capabilities.", "questions": "While the work presents valuable theoretical contributions, significant improvements in code availability, documentation, and practical validation are needed for broader impact and reproducibility.\n\n### 1. Code Release Improvements\n- Release code on popular platforms (GitHub/GitLab)\n- Provide comprehensive documentation\n- Include example applications\n- Add benchmark results reproduction scripts", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- First work to achieve joint generation of LiDAR point clouds and multi-view camera images with cross-modal consistency\n- Novel epipolar-based cross-modal condition module addressing alignment between point clouds and images in different geometric spaces\n- Multi-level controllable generation incorporating text, bounding boxes, images and point clouds as conditions", "weaknesses": "## Limitations in Innovation\n- Weak theoretical foundation for cross-modal consistency, lacking rigorous proof of epipolar-based design effectiveness\n- Incomplete solution for \"spatial ambiguity\", relying solely on epipolar geometry may not fully resolve 3D space ambiguities\n- Limited innovation in generation control, mainly applying existing control methods to new scenarios\n\n## Experimental Analysis\n\n### Required Additional Experiments\n- Evaluation of generated data utility in downstream tasks (3D object detection, trajectory prediction)\n- Need to adopt more advanced transformer architectures like DiT instead of U-Net\n- Consider replacing CLIP with T5 for potentially better performance\n- Address low resolution issues affecting geometric shapes of distant objects (The current resolution is too low. Objects in the distance have no geometric shapes at all. However, the point cloud may have relatively complete and accurate geometric shapes in the distance. A larger resolution should be used, and because the generated image has slightly less geometric shapes in the distance, the effect is not good, simple and practical super-resolution model. It can only increase the resolution but cannot improve the distant geometry.)\n- Investigate video generation capabilities with 3D VAE and 3D attention mechanisms\n- Develop quantitative metrics for cross-modal consistency evaluation\n\n## Ablation Study Analysis\n\n### Missing Ablation Studies\n- Detailed component-wise analysis of cross-modal epipolar condition module\n- Impact assessment of different condition types on generation quality\n- Sensitivity analysis of key hyperparameters\n- Investigation of different training strategies", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)