X-Drive: Cross-modality Consistent Multi-Sensor Data Synthesis for Driving Scenarios

## Abstract

Abstract Recent advancements have exploited diffusion models for the synthesis of either LiDAR point clouds or camera image data in driving scenarios.
Despite their success in modeling single-modality data marginal distribution, there is an under-exploration in the mutual reliance between different modalities to describe complex driving scenes. To fill in this gap, we propose a novel framework, X-Drive , to model the joint distribution of point clouds and multi-view images via a dual-branch latent diffusion model architecture.
Considering the distinct geometrical spaces of the two modalities, X-Drive conditions the synthesis of each modality on the corresponding local regions from the other modality, ensuring better alignment and realism.
To further handle the spatial ambiguity during denoising, we design the cross-modality condition module based on epipolar lines to adaptively learn the cross-modality local correspondence.
Besides, X-Drive allows for controllable generation through multi-level input conditions, including text, bounding box, image, and point clouds. Extensive results demonstrate the high-fidelity synthetic results of X-Drive for both point clouds and multi-view images, adhering to input conditions while ensuring reliable cross-modality consistency. Our code will be made publicly available at https://github.com/yichen928/X-Drive .

# X-Drive: Cross-modality consistent multi-sensor data synthesis for driving scenarios

## 1 Introduction

Autonomous driving vehicles perceive the world with multiple sensors of different kinds, where LiDAR and cameras play crucial roles by capturing point clouds and multi-view images.
They provide complementary geometric measurements and semantic information about the surrounding environment, significantly benefiting tasks such as object detection (Liu et al., ; Xie et al., ), motion planning (Sobh et al., ), scene reconstruction (Huang et al., ; Zhou et al., ), and self-supervised representation learning (Yang et al., ; Xie et al., ).
However, these advancements hinge on access to large amounts of aligned multi-modality data, specifically well-calibrated LiDAR and multi-view camera inputs that describe the same scene.

Scaling up the collection of such high-quality multi-modality data is costly and a non-trivial effort.
The high-quality sensors are expensive, and the calibration process demands intensive human efforts.
Additionally, real-world driving data suffers from the severe long-tailed distribution problem, making it an obstacle for corner case collection such as in extreme weather conditions.
This raises a natural question: Can we use a controllable way to synthesize aligned multi-modality data?

Given the success in other fields (Rombach et al., ; Blattmann et al., ; Liu et al., ), generative models offer a promising solution.
Current research focuses either on synthesizing point clouds (Zyrianov et al., ; Hu et al., ; Ran et al., ) or multi-view images (Gao et al., ; Wang et al., ; Wen et al., ), with limited attention paid to generating multi-modality data.
A simple combination of these single-modality algorithms results in serious cross-modality mismatches in the synthetic scenes (Fig. 1(a)).
Such inconsistencies create ambiguities and even contradictory inputs or supervision signals, thus hindering the performance of downstream tasks.

Cross-modality consistency serves as the key desiderata of multi-modality data generation. However, there are several challenges in the generation of consistent LiDAR and camera data.
Firstly, synthetic point clouds and multi-view images must be spatially aligned in all the local regions since they describe the same driving scene, i.e. the shapes and layouts of both foregrounds and backgrounds must be matched.
Secondly, unlike those 2D pixel-level tasks (Zhang et al., ), point clouds and multi-view images have distinct geometrical spaces and data formats. Multi-view images are represented by RGB values in camera perspective views, while point clouds are XYZ coordinates in the 3D space.
Thirdly, the 3D spatial information is ambiguous during generation for both point clouds and multi-view images without reliable point location or pixel depth in the denoising process.

In this paper, we fill in this gap by proposing X-Drive, a novel framework for the joint generation of LiDAR and camera data, as demonstrated in Fig. 1(b).
We design a dual-branch architecture with two latent diffusion models separately dedicated to the synthesis of point clouds and multi-view images, while a key cross-modality condition module enhances cross-modality consistency between them.
To model the joint distribution and ensure local spatial alignment, we perform the cross-modality conditions locally with only corresponding regions from the other modality considered.
An explicit transform bridges the two different geometrical spaces, converting the cross-modality condition to match each other’s noisy latent space.
To handle the positional ambiguity, we resort to a 3D-aware design based on epipolar lines on range images and multi-view images, allowing the cross-attention module to adaptively determine the cross-modality correspondence without explicit 3D positions.
In consequence, X-Drive is able to exploit existing single-modality data as conditions to seamlessly generate data in the other modality.
Additionally, X-Drive enhances controllability by introducing 3D bounding boxes for geometrical layout control and text prompts for attribute control (e.g. weather and lighting), enabling more flexible, fine-grained, and precise control over both modalities.

Extensive experiments demonstrate the great ability of X-Drive in generating realistic multi-modality sensor data. It notably outperforms previous specialized single-modality algorithms in the quality of both synthetic point clouds and multi-view images.
More importantly, for the first time, it demonstrates reliable cross-modality consistency in synthetic scenes with comprehensive and flexible conditions.
Our contributions are summarized as follows.

- •
We introduce X-Drive, a dual-branch multi-modality latent diffusion framework that, for the first time, enables controllable and reliable synthesis of aligned LiDAR and multi-view camera data.
- •
Our cross-modality epipolar condition module bridges the geometrical gap under spatial ambiguity between point clouds and multi-view images, significantly enhancing modality consistency.
- •
Extensive experimental results demonstrate the effectiveness of X-Drive, with notable MMD (for point cloud generation) and FID (for multi-view generation) improvements, establishing a new state-of-the-art for remarkable cross-modality consistency.

## 2 Related Works

Conditional generation with diffusion models. Diffusion models (Ho et al., ; Dhariwal & Nichol, ; Podell et al., ; Peebles & Xie, ) exhibit remarkable ability in generating diverse images by learning a progressive denoising process. They yield state-of-the-art results in various tasks such as text-to-image generation (Rombach et al., ; Nichol et al., ), text-to-video generation (Singer et al., ; Yu et al., ), and instructional image editing (Brooks et al., ; Meng et al., ). Beyond text condition, several works emerges with the competence in managing additional forms of control signals. Zhang et al. () integrates spatial conditions to a pretrained text-to-image diffusion model via efficient finetuning. Li et al. (); Zheng et al. () condition the image synthesis on the fine-grained geometrical annotations to facilitate downstream tasks like 2D object detection. Liu et al. (); Sargent et al. () introduce 3D-aware diffusion models for novel view synthesis based on single-view image inputs. Unlike these prior work for image domain, our framework simultaneously generates point clouds and multi-view images conditioned on text prompts and 3D bounding boxes.

Cross-modality data generation. Beyond image domain, generative models are extended to the synthesis of multi-modality data. Hu et al. (); Shao et al. () leverage video diffusion model to estimate sequential depths. Bai et al. () formulates various vision tasks as next token prediction by extracting diverse visual information as unified visual tokens. The cross-modality synthesis between videos and audios also draws great attention. Efforts are devoted to the video-to-audio (Zhu et al., ), audio-to-video (Chatterjee & Cherian, ), bi-directional (Chen et al., ; Hao et al., ), and joint multi-modality (Ruan et al., ) generation. However, in all the above cases, there exists clear cross-modality alignments, such as the pixel-to-pixel spatial correspondence for depth estimation and frame-to-frame temporal alignment between audios and videos. In contrast, point clouds and multi-view images are in different geometrical spaces without clear one-to-one local correlation between noisy multi-modality samples.

Generation of vehicle sensor data. Panoramic driving environment is usually perceived by the multi-view cameras and LiDARs on the ego-vehicle. Alternative to the laborious collection and annotation process, many works resort to the fast-growing generative models for the synthesis of either camera (Swerdlow et al., ; Yang et al., ; Gao et al., ) or LiDAR (Zyrianov et al., ; Hu et al., ; Zyrianov et al., ) sensor data. For multi-view images (Yang et al., ; Wang et al., ; Gao et al., ) or videos (Wen et al., ; Lu et al., ), previous work leverages pretrained image diffusion models (Rombach et al., ) by incorporating extra inter-view modules to ensure the multi-view consistency. For point clouds, Zyrianov et al. (); Ran et al. (); Hu et al. () adapt latent diffusion models for the generation of range images (Fan et al., ; Li et al., ) since this range-view representation shares a similar format with RGB images. However, all above methods concentrate on the single-modality data, either LiDAR or camera sensor. Despite various control signals, there exists no guarantee for the cross-modality consistency between point clouds and multi-view images generated by independent single-modality models. To this end, we propose to synthesize consistent multi-modality data in a joint manner.

## 3 Preliminary

Latent Diffusion Models. Diffusion models (Ho et al., ; Sohl-Dickstein et al., ) learn to approximate a data distribution $p(\mathbf{x}_{0})$ by iteratively denoising a random Gaussian noise for $T$ steps. Typically, diffusion models construct a diffused input $\mathbf{x}_{t}$ through a forward process, which gradually adds Gaussian noise to the data according to a variance schedule $\beta_{1},\dots,\beta_{T}$.

$$ $q(\mathbf{x}_{1:T}|\mathbf{x}_{0})=\prod_{t=1}^{T}q(\mathbf{x}_{t}|\mathbf{x}_ {t-1}),\quad q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\mathcal{N}(\mathbf{x}_{t}; \sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I}).$ (1) $$

Then, the reverse process learns to recover the original inputs by fitting $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$.

$$ $p_{\theta}(\mathbf{x}_{0:T})=p(\mathbf{x}_{T})\prod_{t=1}^{T}p_{\theta}( \mathbf{x}_{t-1}|\mathbf{x}_{t}),\quad p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_ {t})=\mathcal{N}(\mathbf{x}_{t-1};\mu_{\theta}(\mathbf{x}_{t},t),\Sigma_{ \theta}(\mathbf{x}_{t},t)).$ (2) $$

Latent diffusion models (Rombach et al., ) perform the diffusion process on the latent space instead of the input space to handle the high dimensional data. Specifically, it maps input $\mathbf{x}$ with an encoder $\mathcal{E}$ into the latent space as $\mathbf{z}=\mathcal{E}(\mathbf{x})$. The latent code $\mathbf{z}$ can be reconstructed to the input as $\mathbf{\hat{x}}=\mathcal{D}(\mathbf{z})$ through a decoder $\mathcal{D}$. The forward and reverse processes of latent diffusion models are similar with the original diffusion models by substituting $\mathbf{x}$ with latent $\mathbf{z}$ in Eq. 1 and Eq. 2.

Range Image Representation. Compatible with the sampling process of LiDAR sensor, range image $\mathcal{R}\in\mathbb{R}^{H_{r}\times W_{r}\times 2}$ is a dense rectangular representation for LiDAR point clouds, where $H_{r}$ and $W_{r}$ are the numbers of rows and columns with one channel representing the range and the other representing intensity. The rows reflect the laser beams while the columns indicate the yaw angles. For each point with Cartesian coordinate $(x,y,z)$, it can be transformed to the spherical coordinates $(r,\theta,\phi)$ through the following projection.

$$ $r=\sqrt{x^{2}+y^{2}+z^{2}},\quad\theta=\arctan(y,x),\quad\phi=\arctan(z,\sqrt{ x^{2}+y^{2}})$ (3) $$

where $r$ is the range, $\theta$ is the inclination, and $\phi$ is the azimuth. The range image is produced by quantizing the $\theta$ and $\phi$ with factors $s_{\theta}$ and $s_{\phi}$. For each point with spherical coordinate $(r_{i},\theta_{i},\phi_{i})$, we have $\mathcal{R}(\lfloor\theta_{i}/s_{\theta}\rfloor,\lfloor\phi_{i}/s_{\phi}
\rfloor)=(r_{i},i_{i})$. The range $r_{i}$ and intensity $i_{i}$ are normalized to $(0,1)$.

Figure: Figure 2: Overview of our proposed X-Drive framework. We design a dual-branch diffusion model architecture to generate multi-modality data. Cross-modality epipolar condition modules (Fig. 3) are inserted between branches to enhance the cross-modality consistency.
Refer to caption: x3.png

## 4 Methodology

In this section, we present our novel framework, X-Drive, for joint generation of point clouds and multi-view images. The overall architecture is depicted in Fig. 2. We extend the vanilla diffusion models to modeling the joint distribution of multi-modality data in Sec. 4.1. Practically, it is achieved by our proposed framework with dual-branch diffusion models in Sec. 4.2. For the cross-modality consistency, we propose a cross-modality epipolar condition module in Sec. 4.3 . Our method can also be tailored for cross-modality conditional generation in a zero-shot manner as in Sec. 4.4.

### 4.1 Joint Multi-modality Generation

Beyond single-modality diffusion models in Sec. 3, we give a formulation to our proposed X-Drive for multi-modality data generation. It aims to approximate a joint distribution for paired data $(\mathbf{r}_{0},\mathcal{C}_{0})$ that describes a specific driving scene, where $\mathbf{r}_{0}$ is the range image representation of point clouds and $\mathcal{C}_{0}=\{\mathbf{x}_{0}^{v}\}_{v=1}^{V}$ denotes multi-view images from $V$ cameras at different perspective views.

Given their similar rectangular latent formats, we set a shared noise schedule $\beta_{1},\dots,\beta_{T}$ for range images and multi-view images. The forward processes of either range image or multi-view images only depend on its own modality, equivalent with independent forward processes for each modality.

$$ $q(\mathbf{r}_{t}|\mathbf{r}_{t-1})=\mathcal{N}\left(\mathbf{r}_{t};\sqrt{1- \beta_{t}}\mathbf{r}_{t-1},\beta_{t}\mathbf{I}\right),\quad q(\mathcal{C}_{t}| \mathcal{C}_{t-1})=\mathcal{N}\left(\{\mathbf{x}_{t}^{v}\}_{v};\{\sqrt{1-\beta _{t}}\mathbf{x}^{v}_{t-1}\}_{v},\left\{\beta_{t}\mathbf{I}\right\}_{v}\right)$ . (4) $$

In contrast, the reverse process exploits models $\epsilon_{\theta_{r}},\epsilon_{\theta_{c}}$ to predict the noise $\bm{\epsilon}_{r}$ and $\bm{\epsilon}_{c}$ added to the range image and multi-view images separately. Given the requirements for cross-modality consistency, the noise prediction for each modality should take each other into consideration, so the reverse process for either range image or multi-view images would depend on both modalities.

$$ $\displaystyle p_{\theta_{r}}(\mathbf{r}_{t-1}|\mathbf{r}_{t},\mathcal{C}_{t})= \mathcal{N}(\mathbf{r}_{t-1};\mu_{\theta_{r}}(\mathbf{r}_{t},\mathcal{C}_{t},t ),\Sigma_{\theta_{r}}(\mathbf{r}_{t},\mathcal{C}_{t},t)),$ (5) $\displaystyle p_{\theta_{c}}(\mathcal{C}_{t-1}|\mathcal{C}_{t},\mathbf{r}_{t}) =\mathcal{N}(\mathcal{C}_{t-1};\mu_{\theta_{c}}(\mathcal{C}_{t},\mathbf{r}_{t} ,t),\Sigma_{\theta_{c}}(\mathcal{C}_{t},\mathbf{r}_{t},t)).$ $$

Intuitively, we can implement two diffusion models as $\epsilon_{\theta_{r}}$ and $\epsilon_{\theta_{c}}$ separately for range images and multi-view images, equipped with encoders $c_{CR}$ and $c_{RC}$ for the other modality as extra conditions.

$$ $\hat{\bm{\epsilon}}_{r}=\epsilon_{\theta_{r}}(\mathbf{r}_{t},c_{CR}(\mathcal{C }_{t},t),t),\quad\hat{\bm{\epsilon}}_{c}=\epsilon_{\theta_{c}}(\mathcal{C}_{t} ,c_{RC}(\mathbf{r}_{t},t),t).$ (6) $$

Instead of training separate condition encoders, we can directly treat diffusion models themselves as strong encoders for each modality. Thus, $\epsilon_{\theta_{c}}$ and $\epsilon_{\theta_{r}}$ are adapted as the condition encoder $\epsilon^{\prime}_{\theta_{c}}(\mathcal{C}_{t},t)$ and $\epsilon^{\prime}_{\theta_{r}}(\mathbf{r}_{t},t)$ in each block. Compared to text-to-image condition, a notable characteristic of LiDAR-camera mutual condition lies in the local spatial correspondence. For example, a car captured by LiDAR should share the consistent shape and location with the same car captured by cameras but has little direct relationship with other parts in the images. Due to their different geometrical spaces, we should rely on a spatial transformation $\mathcal{T}(\cdot)$ to explicitly align the geometrical space for cross-modality correspondence. Concretely, $\mathcal{T}_{CR}$ transforms the multi-view image conditions to the range image space, while $\mathcal{T}_{RC}$ transforms the range image conditions to the multi-view image space. As a result, the condition encoders $c_{CR},c_{RC}$ can be formulated as follows.

$$ $c_{CR}(\mathcal{C}_{t},t)=\mathcal{T}_{CR}(\epsilon^{\prime}_{\theta_{c}}( \mathcal{C}_{t},t)),\quad c_{RC}(\mathbf{r}_{t},t)=\mathcal{T}_{RC}(\epsilon^{ \prime}_{\theta_{r}}(\mathbf{r}_{t},t)).$ (7) $$

In this case, we rewrite the denoising models in Eq. 6 as follows.

$$ $\hat{\bm{\epsilon}}_{r}=\epsilon_{\theta_{r}}(\mathbf{r}_{t},\mathcal{T}_{CR}( \epsilon^{\prime}_{\theta_{c}}(\mathcal{C}_{t},t)),t),\quad\hat{\bm{\epsilon}} _{c}=\epsilon_{\theta_{c}}(\mathcal{C}_{t},\mathcal{T}_{RC}(\epsilon^{\prime}_ {\theta_{r}}(\mathbf{r}_{t},t)),t).$ (8) $$

They can be trained with a joint multi-modality objective function $\mathcal{L}_{DM-M}$.

$$ $\mathcal{L}_{DM-M}=\mathcal{L}_{DM-R}+\mathcal{L}_{DM-C},\quad\mathcal{L}_{DM- R}=\mathbb{E}_{r,t,\epsilon_{r}}||\epsilon_{r}-\hat{\epsilon}_{r}||_{2}^{2}, \quad\mathcal{L}_{DM-C}=\mathbb{E}_{\mathcal{C},t,\epsilon_{c}}||\epsilon_{c}- \hat{\epsilon}_{c}||_{2}^{2}$ . (9) $$

### 4.2 Dual-Branch Joint Generation Framework

Our framework is developed following Eq. 8, which consists of two denoising models, i.e., $\epsilon_{\theta_{r}}(\cdot)$ and $\epsilon_{\theta_{c}}(\cdot)$ respectively tailored for range images (Sec. 4.2.1) and multi-view images (Sec. 4.2.2).

#### 4.2.1 Diffusion Model for Range Images

Architecture.
We adapt existing latent diffusion model (LDM) (Rombach et al., ) to learning the distribution of range image $\mathbf{r_{0}}\in\mathbb{R}^{H_{r}\times W_{r}\times 2}$ given its similar representation format with RGB images. In the first stage, the range image is compressed into latent feature $\mathbf{z}_{0}^{r}\in\mathbb{R}^{h_{r}\times w_{r}\times c_{r}}$ by a downsampling factor $f_{r}=H_{r}/h_{r}=W_{r}/w_{r}$. In the second stage, we train an LDM for range images from scratch to learn the latent distribution of $\mathbf{z}_{0}^{r}$. Range image reflects the panoramic view of the ego-vehicle, so its left-most and right-most sides are connected. Considering this constraint, we follow LiDARGen (Zyrianov et al., ) and RangeLDM (Hu et al., ) to replace all the convolutions in both VAE and LDM with horizontally circular convolutions (Schubert et al., ) where the left and right sides of the range image are treated as neighbors.

In the first stage, we follow the standard protocol to train the VAE including both encoder $\mathcal{E}_{r}$ and decoder $\mathcal{D}_{r}$ by maximizing the ELBO (Rezende et al., ). In addition, we attach an adversarial discriminator (Isola et al., ; Rombach et al., ) to mitigate the blurriness brought by the reconstruction loss. In the second stage, the LDM conditions the synthesis of range images on the cross-modality information from the multi-view image branch (elaborated in Sec. 4.3) as well as the text prompt and 3D boxes. The model is trained with loss function in Eq. 9.

Range-view bounding box condition.
Given a 3D bounding box $(\mathbf{b}_{i},c_{i})$, $\mathbf{b}_{i}=\{(x_{j},y_{j},z_{j})\}_{j=1}^{8}\in\mathbb{R}^{8\times 3}$ represents the 3D coordinates of box corners and $c_{i}$ denotes its semantic category. We project each box corner to the range view using Eq. 3 as $\mathbf{b}_{i}^{r}=\{(r_{j},\theta_{j},\phi_{j})\}_{j=1}^{8}$. These range-view corner coordinates are passed through a Fourier embedder. Then, the concatenated position embeddings of eight corners are encoded by $\text{MLP}_{p}^{r}$ as $\mathbf{h}_{p,i}^{r}$. For category label $c_{i}$, we follow (Isola et al., ; Gao et al., ) to pool the CLIP embedding (Radford et al., ) of the category name as $\mathbf{h}_{l,i}^{r}$. The box and label embeddings are combined and encoded by another $\text{MLP}_{b}^{r}$ as a hidden vector $\mathbf{h}^{r}_{b,i}$.

$$ $\mathbf{h}^{r}_{b,i}=\text{MLP}_{b}^{r}([\mathbf{h}^{r}_{l,i},\mathbf{h}_{p,i} ^{r}]),\quad\mathbf{h}^{r}_{l,i}=\text{Avg}(\text{CLIP}(c_{i})),\quad\mathbf{h }_{p,i}^{r}=\text{MLP}_{p}^{r}(\text{Concat}(\text{Foruier}(\mathbf{b}_{i}^{r} )))$ (10) $$

We concatenate box hidden vectors $\{\mathbf{h}^{r}_{b,i}\}_{i}$ from this range-view box encoder with text prompt hidden vectors, which guide the synthesis the latent diffusion model via a cross-attention module.

#### 4.2.2 Diffusion Model for Multi-view Images

Architecture. To model the distribution of multi-view images $\mathcal{C}_{0}=\{\mathbf{x}_{0}^{v}\}_{v=1}^{V},\mathbf{x}_{0}^{v}\in\mathbb{
R}^{H_{c}\times W_{c}\times 3}$, we tailor the latent diffusion model by introducing extra bounding box and inter-view conditions apart from the cross-modality signals from range images (details in Sec. 4.3). Following standard LDM (Rombach et al., ), each image $\mathbf{x}_{0}^{v}$ is first compressed by a pretrained VAE to the latent space as $\mathbf{z}_{0}^{c,v}$. In the training process, we sample the Gaussian noise for each view independently and the denoising model is trained with the objective function in Eq. 9.

Multi-view bounding box condition. The 3D box condition module is similar with Sec. 4.2.1, but the box positions are transformed to the perspective view of each camera. For each 3D bounding box $(\mathbf{b}_{i},c_{i})$, we project its 3D corner coordinates $\mathbf{b}_{i}=\{(x_{j},y_{j},z_{j})\}_{j=1}^{8}$ to corresponding camera perspective view as $\mathbf{b}_{i}^{c}=\{(u_{j},v_{j},d_{j})\}_{j=1}^{8}$, where $u_{j},v_{j}$ are the corner location in the pixel coordinate and $d_{j}$ is the depth. For each camera view $v$, its synthesis is only conditioned on the 3D bounding boxes with at least one corner projected into the range of its perspective view image. Then, we formulate this perspective-view box encoder similar with Eq. 10.

$$ $\mathbf{h}^{c}_{b,i}=\text{MLP}_{b}^{c}([\mathbf{h}^{c}_{l,i},\mathbf{h}_{p,i} ^{c}]),\quad\mathbf{h}^{c}_{l,i}=\text{Avg}(\text{CLIP}(c_{i})),\quad\mathbf{h }_{p,i}^{c}=\text{MLP}_{p}^{c}(\text{Concat}(\text{Fourier}(\mathbf{b}_{i}^{c} )))$ (11) $$

For each camera view, the box embeddings $\{\mathbf{h}^{c}_{b,i}\}_{i}$ are concatenated with the text prompt hidden vectors, which together guide the image generation through a cross-attention module.

Inter-view condition. It is critical to enhance the consistency across different views in the generation of multi-view images $\mathcal{C}_{0}=\{\mathbf{x}_{0}^{v}\}_{v=1}^{V}$, so we inject inter-view cross-attention module to condition the generation of each camera view $v$ on its left and right adjacent views $vl$ and $vr$. Given the small overlapping between field-of-views of adjacent cameras, we split each image latent $\mathbf{z}_{in}^{c,v}$ into two halves horizontally. For each view $v$, its left half $\mathbf{z}_{in,l}^{c,v}$ attends to the right half $\mathbf{z}_{in,r}^{c,vl}$ of its left neighbor view $vl$, while its right half $\mathbf{z}_{in,r}^{c,v}$ depends on the left half $\mathbf{z}_{in,l}^{c,vr}$ of its right neighbor $vr$.

$$ $\mathbf{z}_{out}^{c,v}=\mathbf{z}_{in}^{c,v}+\tanh{(\alpha_{v})}\cdot\text{ Concat}_{width}\left(\left[\text{CrossAttn}(\mathbf{z}_{in,l}^{c,v},\mathbf{z} _{in,r}^{c,vl});\text{CrossAttn}(\mathbf{z}_{in,r}^{c,v},\mathbf{z}_{in,l}^{c, vr})\right]\right)$ (12) $$

where $\alpha_{v}$ is a zero-initialization gate (Zhang et al., ) for stable optimization. Compared to full cross-attention in MagicDrive (Gao et al., ), our split strategy significantly reduces the per-scene GPU memory cost of inter-view condition from 11GB to 3GB with better multi-view consistency.

Figure: Figure 3: Cross-modality epipolar condition module. We perform mutual conditions locally between LiDAR and camera modalities based on epipolar lines on multi-view image and range image latents.
Refer to caption: x4.png

### 4.3 Cross-Modality Epipolar Condition Module

The key to multi-modality data synthesis is to boost the cross-modality consistency, which potentially relies on cross-modality conditions. Ideally, there is an explicit point-to-pixel correspondence between point clouds and multi-view images through camera projection. However, in the denoising process, we are not aware of either the range values of range images or the depths of the multi-view images. Instead, we propose to warp the local features from range images and multi-view images based on epipolar lines through spatial transforms $\mathcal{T}_{RC}$ and $\mathcal{T}_{CR}$ (Eq. 7) to provide control signals for the synthesis of the other modality, illustrated in Fig. 3.

Camera-to-LiDAR condition.
For each position $(\phi,\theta)$ on the range image latent $\mathbf{z}_{in}^{r}$, we sample $R$ points along the range axis following
the linear-increasing discretization (LID) (Reading et al., ; Liu et al., ) with range value $r_{k},k=1,2,\dots,R$. We transform these points $\left\{(\phi,\theta,r_{k})\right\}_{k=1}^{R}$ to their 3D coordinates $\{(x_{k},y_{k},z_{k})\}_{k=1}^{R}$.

$$ $x_{k}=r_{k}\cdot\cos(\phi)\cdot\sin(\theta),\quad y_{k}=r_{k}\cdot\cos(\phi) \cdot\cos(\theta),\quad z_{k}=r_{k}\cdot\sin(\phi).$ (13) $$

They can be projected to the camera perspective view $v$ with the camera parameters $(\mathbf{R}_{v},\mathbf{t}_{v},\mathbf{K}_{v})$ as

$$ $d_{k}\begin{bmatrix}u_{k}&v_{k}&1&\end{bmatrix}^{T}=\mathbf{K}_{v}\left( \mathbf{R}_{v}\begin{bmatrix}x_{k}&y_{k}&z_{k}\end{bmatrix}^{T}+\mathbf{t}_{v} \right),\quad k=1,2,\dots,R$ (14) $$

where $(u_{k},v_{k}),k=1,2,\dots,R$ are the pixel coordinates for the $R$ points along the LiDAR ray. They form the epipolar line on the camera image $\mathbf{x}^{v}$ corresponding to position $(\phi,\theta)$ of the LiDAR range image. We bilinearly sample the corresponding local features from the image latent $\mathbf{z}_{in}^{c,v}$ of view $v$ at location $(u_{k},v_{k})$. If a 3D point $(x_{k},y_{k},z_{k})$ is projected into more than one camera view, we would simply adopt the average local features. Extra Fourier embedding of $r_{k}$ is added to each sampled feature as an indicator for the range value along the ray. As a result, the entire spatial transform $\mathcal{T}_{CR}$ makes the camera image features spatially aligned to the range image features.

$$ $\mathcal{T}_{CR}(\mathbf{z}_{in}^{c,v})_{(\phi,\theta)}=\left\{\text{Bilinear- Sample}(\mathbf{z}_{in}^{c,v};(u_{k},v_{k}))+\text{MLP}_{r}\left(\text{Fourier }(r_{k})\right)\right\}_{k=1}^{R}.$ (15) $$

We apply a cross-attention module to condition the range image feature $\mathbf{z}_{in,(\phi,\theta)}^{r}$ at coordinate $(\phi,\theta)$ on the corresponding transformed local image features $\mathcal{T}_{CR}(\mathbf{z}_{in}^{c})_{(\phi,\theta)}$ on the epipolar line.

$$ $\mathbf{z}_{out,(\phi,\theta)}^{r}=\mathbf{z}_{in,(\phi,\theta)}^{r}+\tanh( \alpha_{cr})\cdot\text{CrossAttn}\left(\mathbf{z}_{in,(\phi,\theta)}^{r}, \mathcal{T}_{CR}(\mathbf{z}_{in}^{c,v})_{(\phi,\theta)}\right)$ (16) $$

where $\alpha_{cr}$ is a zero-initialization gate for the camera-to-LiDAR condition. The model learns the adaptive local correspondence between range image and multi-view images under range ambiguity.

LiDAR-to-camera condition. Inversely, we condition the synthesis of multi-view images on the LiDAR range images using a similar module. For each position $(u,v)$ on the camera image latent $\mathbf{z}_{in}^{c,v}$ from view $v$, $D$ points are sampled at different depths following the linear-increasing discretization with depth value $d_{k},k=1,2,\dots,D$. Each point $(u,v,d_{k})$ in the pixel coordinate is transformed to 3D coordinate $(x_{k},y_{k},z_{k})$ via the corresponding camera parameters $(\mathbf{R}_{v},\mathbf{t}_{v},\mathbf{K}_{v})$.

$$ $\begin{bmatrix}x_{k}&y_{k}&z_{k}\end{bmatrix}^{T}=\mathbf{R}_{v}^{T}\left( \mathbf{K}_{v}^{-1}\begin{bmatrix}u_{k}\cdot d_{k}&v_{k}\cdot d_{k}&d_{k}\end{ bmatrix}^{T}-\mathbf{t}_{v}\right),\quad k=1,2,\dots,D$ (17) $$

The 3D point coordinate is then projected to the LiDAR range view using Eq. 3 as $(\phi_{k},\theta_{k},r_{k})$. The $D$ points $(\phi_{k},\theta_{k}),k=1,2,\dots,D$ form the epipolar line on the LiDAR range image corresponding to the pixel coordinate $(u,v)$ of camera view $v$. The local range image features are extracted at each coordinate $(\phi_{k},\theta_{k})$ from range image latent $\mathbf{z}_{in}^{r}$ through bilinear sampling with additional Fourier embedding for the depth value $d_{k},k=1,2,\dots,D$ attached. We can write the spatial transform $\mathcal{T}_{RC}$ aligning the range view features to multi-view image features as follows.

$$ $\mathcal{T}_{RC}(\mathbf{z}_{in}^{r})_{(u,v)}=\left\{\text{Bilinear-Sample}( \mathbf{z}_{in}^{r};(\theta_{k},\phi_{k}))+\text{MLP}_{d}\left(\text{Fourier}( d_{k})\right)\right\}_{k=1}^{D}.$ (18) $$

Similar with camera-to-LiDAR condition, a cross-attention module conditions the local camera image feature $\mathbf{z}_{out,(u,v)}^{c,v}$ at coordinate $(u,v)$ on spatially aligned range image features $\mathcal{T}_{RC}(\mathbf{z}_{in}^{c,v})_{(u,v)}$.

$$ $\mathbf{z}_{out,(u,v)}^{c,v}=\mathbf{z}_{in,(u,v)}^{c,v}+\tanh(\alpha_{rc}) \cdot\text{CrossAttn}\left(\mathbf{z}_{in,(u,v)}^{c,v},\mathcal{T}_{RC}( \mathbf{z}_{in}^{r})_{(u,v)}\right)$ (19) $$

where $\alpha_{rc}$ is a zero-initialization gate. The cross-attention module adaptively learns the local reliance of camera multi-view images on LiDAR range images without explicit depth cues.

### 4.4 Cross-Modality Conditional Generation

Besides multi-modality joint generation, X-Drive can work as a camera-to-LiDAR or LiDAR-to-camera conditional generation model in a zero-shot manner without targeted training. Given either one of ground-truth LiDAR range image $\textbf{r}_{0}$ or multi-view images $\mathcal{C}_{0}$, we change Eq. 8 separately for camera-to-LiDAR or LiDAR-to-camera generation to synthesize the other modality data.

$$ $\hat{\bm{\epsilon}}_{r}=\epsilon_{\theta_{r}}(\mathbf{r}_{t},\mathcal{T}_{CR}( \epsilon^{\prime}_{\theta_{c}}(\mathcal{C}_{0},0)),t),\quad\hat{\bm{\epsilon}} _{c}=\epsilon_{\theta_{c}}(\mathcal{C}_{t},\mathcal{T}_{RC}(\epsilon^{\prime}_ {\theta_{r}}(\mathbf{r}_{0},0)),t).$ (20) $$

In this case, the trained diffusion model for the input modality serves as a strong encoder for the condition to enhance the adherence of generation to the input LiDAR or camera conditions.

## 5 Experiments

### 5.1 Experimental Setups

Dataset. We evaluate our method using nuScenes dataset (Caesar et al., ). It provides point clouds from a 32-beam LiDAR and multi-view images from 6 cameras. We follow the official setting to employ 700 driving scenes for training and 150 scenes for validation. The generation of X-Drive is conditioned on bounding boxes from 10 semantic classes as well as scene descriptions.

Evaluation Metrics. For joint multi-modality data synthesis, we evaluate both single-modality data realism and cross-modality consistency. For image modality, we report Fréchet Inception Distance (FID) for image synthesis realism. We also utilize View Consistency Score (VSC) (Swerdlow et al., ) to evaluate multi-view consistency, which reflects the keypoints correspondence on overlapping regions between adjacent camera views (Fig. 5), and we normalize the VSC of real images to $1.0$ for clarity. For point clouds modality, we measure the distribution gap between synthetic and real point clouds with Maximum Mean Discrepancy (MMD) and Jensen-Shannon divergence (JSD) following RangeLDM (Hu et al., ). For cross-modality consistency, we propose a novel metric called Depth Alignment Score (DAS). We project the synthetic point clouds to multi-view images as sparse depth values and also estimate the image depth using DepthAnythingV2 (Yang et al., ). The mean absolute error between the projected and estimated disparities is used as DAS metric.

Baselines. To our best knowledge, we are the first to work on the joint generation of point clouds and multi-view images. For single-modality generation, we compare with state-of-the-art generation algorithms for point clouds, i.e. LiDAR VAE (Caccia et al., ), LiDARGen (Zyrianov et al., ), RangeLDM (Hu et al., ), and multi-view images, i.e. BEVGen (Swerdlow et al., ), BEVControl (Yang et al., ), MagicDrive (Gao et al., ), with released code or quantitative results on nuScenes dataset. Furthermore, we combine MagicDrive (Gao et al., ) and RangeLDM (Hu et al., ) respectively for images and point clouds as a multi-modality baseline.

Training Setup. Our X-Drive has a dual-branch architecture. We utilize the Stable-Diffusion V1.5 pretrained weight to initialize the multi-view image branch with other newly added parameters randomly initialized. We follow a three-stage pipeline for the training. In the first stage, the VAE is trained from scratch for range images. In the second stage, we train the latent diffusion model in point clouds branch without cross-modality conditions using the frozen VAE. Finally, in the last stage, the entire multi-modality model is trained in an end-to-end manner with only the Stable-Diffusion V1.5 parts frozen. We follow MagicDrive (Gao et al., ) and RangeLDM (Hu et al., ) to synthesize $224\times 400$ multi-view camera images and $32\times 1024$ point cloud range images.

**Table 1: Quantitative comparison with driving data generation algorithms. For each column, the best value is highlighted by bold, and the second best one is denoted by underline.**
| Modality | Method | Image quality | Point clouds quality | Cross-modality |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| FID $\downarrow$ | VSC $\uparrow$ | MMD $\downarrow$ | JSD $\downarrow$ | DAS $\downarrow$ |  |  |
| C | BEVControl | 25.54 | - | - | - | - |
| BEVGen | 24.85 | 0.439 | - | - | - |  |
| MagicDrive | 16.20 | 0.633 | - | - | - |  |
| L$\rightarrow$C | X-Drive (ours) | 16.01 | 0.721 | - | - | - |
| L | LiDAR VAE | - | - | $1.1\times 10^{-3}$ | - | - |
| LiDARGen | - | - | $1.9\times 10^{-3}$ | 0.160 | - |  |
| RangeLDM | - | - | $1.9\times 10^{-4}$ | 0.054 | - |  |
| C$\rightarrow$L | X-Drive (ours) | - | - | $\mathbf{1.0\times 10^{-4}}$ | 0.052 | - |
| C+L | MagicDrive + RangeLDM | - | - | - | - | 2.32 |
| X-Drive (ours) | 17.37 | 0.675 | $\underline{1.2\times 10^{-4}}$ | 0.052 | 1.69 |  |

### 5.2 Quantitative Results

We report the quantitative results for both joint generation of aligned point clouds and multi-view images as well as conditional LiDAR-to-camera or camera-to-LiDAR cross-modality generation.

Joint multi-modality generation. As shown in Tab. 1, in comparison to specialized single-modality generation methods, X-Drive, as a multi-modality algorithm, achieves comparable or even better quality in both synthetic point clouds and multi-view images. Our FID metric is outoerformed by MagicDrive (Gao et al., ), since we do not have map condition which offers strong control signals. Unlike simple combination of single-modality methods, we can generate point clouds and images with cross-modality alignment, reflected by our superior DAS metric, thanks to our proposed cross-modality epipolar condition module. Extra qualitative visualizations are shown in Appendix C.

Conditional cross-modality generation. As mentioned in Sec. 4.4, X-Drive can also work as a LiDAR-to-camera or camera-to-LiDAR conditional generation model. In Tab. 1, for single-modality data generation, X-Drive can outperform previous baselines for both point clouds and multi-view images, demonstrating the flexibility of our proposed algorithm for cross-modality data synthesis.

**Table 2: Object-level control.**
| Methods | mAP | NDS |
| --- | --- | --- |
| Oracle | 70.5 | 72.8 |
| MagicDrive | 65.2 | 69.2 |
| X-Drive (ours) | 65.4 | 69.6 |

Object-level controllable generation. X-Drive can adhere to the 3D bounding box conditions in the generative process. Since previous LiDAR generation methods do not show this ability (Hu et al., ; Ran et al., ), we only compare our methods with multi-view image generation algorithm Gao et al. (). For fair comparison with single-modality method, we employ synthetic multi-view images and real point clouds from nuScenes validation set to run a pretrained SparseFusion (Xie et al., ) model, which is an advanced multi-sensor object detection algorithm. Tab. 2 shows X-Drive outperforms MagicDrive Gao et al. () w.r.t. the object-level fidelity.

Figure: Figure 4: Cross-modality consistency qualitative results for multi-modality generation and conditional cross-modality generation. Colors of point clouds refer to different depths. Well-matched regions between point clouds and multi-view images are highlighted with red circles.
Refer to caption: x5.png

### 5.3 Qualitative Analysis

Cross-modality consistency.
We show some multi-modality generation examples to exhibit the cross-modality consistency qualitatively. As in Fig. 4, the projected point clouds overlap with multi-view image contents properly for both foregrounds and backgrounds. We also show the results of conditional camera-to-LiDAR and LiDAR-to-camera generation in Fig. 4, where the synthetic data adheres to the cross-modality LiDAR or camera conditions well.

Figure: Figure 5: Key points correspondence between adjacent synthetic camera images. Our proposed method (bottom) can bring higher multi-view consistency than the baseline (top).
Refer to caption: extracted/5972524/figures/correspondence.png

Multi-view consistency.
In our multi-modality generation, point clouds provide a natural 3D-aware geometrical guidance for the multi-view images, benefiting the multi-view consistency. In Fig. 5, we visualize the keypoints matching results (Sun et al., ) between adjacent synthetic camera images, where better correspondence is witnessed for X-Drive than MagicDrive (Gao et al., ).

Scene-level and object-level control.
Given the text and bounding box conditions, X-Drive generate diverse results using different control signals, as illustrated in Fig. 6. For scene-level control, we can target for various lighting and weathers through the text prompts, while synthetic images still remain realistic and adhere to the object layouts. For object-level control, we can edit the scene by deleting objects or inserting objects at desired locations with specific sizes and semantic categories.

Figure: Figure 6: Scene-level and object-level controllable generation by changing the input conditions.
Refer to caption: x6.png

Figure: Figure 7: Effect of CFG scales.
Refer to caption: x7.png

### 5.4 Ablation Studies

Due to limited computational resources, we employ shorter training schedule for ablation studies.

Cross-modality condition. The cross-modality condition is critical for cross-modality consistency. As shown in Tab. 3, removing this module significantly hurts the cross-modality alignment although the shared bounding boxes and text prompts still provide some constraints. Given the spatial ambiguity, we resort to cross-attention along epipolar lines to adaptively learn the local correspondence in Eq. 16 and Eq. 19. Instead, simple averaging along epipolar lines instead of cross-attention is quite noisy and poses severe damage to the point clouds quality and cross-modality consistency.

Bounding box and text prompt input conditions. Although X-Drive also supports unconditional generation, these input conditions provide multi-level control signals to generate diverse outputs. Tab. 3 shows that text and box conditions improve the quality of synthetic point clouds and images.

Classifier-free guidance. We employ classifier-free guidance (CFG) during inference for box and text conditions. Illustrated in Fig. 7, it improves the realism of synthetic data. However, higher CFG scale would hurt the image and point cloud quality by increasing the contrast and sharpness.

## 6 Conclusions

We present X-Drive, a novel framework for the multi-modality generation of aligned LiDAR point clouds and multi-view camera images. By leveraging a dual-branch diffusion model, X-Drive captures the joint distribution of multi-modality data, ensuring mutual dependence between modalities. A key feature is the cross-modality condition module, which uses epipolar lines on LiDAR range images and camera views to address spatial ambiguity during the denoising process. Additionally, X-Drive supports multi-level control, enabling synthesis based on text prompts and object bounding boxes. Extensive experiments show that X-Drive generates high-quality, geometrically and semantically aligned point clouds and images, maintaining fidelity to the input conditions while accurately describing the same driving scene together.

##### Acknowledegement

This work was supported in part by Berkeley DeepDrive and NSF Grant. 2235013.

## Appendix A Implementation Details

In this part, we explain details in the implementation of X-Drive including network architecture, training schedule, and evaluation metrics. Our code will be made publicly available upon the acceptance of this paper.

##### Network architecture.

Given the small height dimension of LiDAR range images and limited data amount, we employ smaller sizes for the VAE and latent diffusion model (LDM) for LiDAR range images. Specifically, the VAE includes three encoder blocks and three decoder blocks with channels $[32,64,128]$ separately. There are downsampling modules in the first two encoder blocks. The latent $\mathbf{z}^{r}$ for range image has a size of $8\times 256\times 8$. The range image LDM is composed of three downsampling blocks, one middle block, and three upsampling blocks with attention modules in all the blocks. The channel numbers for three downsampling and upsampling blocks are $[128,256,512]$ respectively. The cross-modality epipolar condition modules are inserted between each pair of corresponding blocks in point cloud and multi-view image diffusion model branches. For these seven cross-modality modules, we set the sampling number $R$ and $D$ along range and depth axes as $[24,24,12,6,12,24,24]$ separately with both maximal depth and range as $60m$.

##### Training schedule.

The training of X-Drive on nuScenes dataset (Caesar et al., ) includes three stages. Since there are no publicly available pretrained weights for LiDAR diffusion models, we train the VAE and LDM for LiDAR range image in the first two stages separately. Afterwards, we train the entire multi-modality generation framework together. Before training, we initialize our multi-view image diffusion model branch with pretrained Stable-Diffusion V1.5 weights, while all the other modules except the zero-initialization gates are initialized randomly. In all the stages, our model is trained using NVIDIA RTX A6000 GPUs.

In the first stage, VAE for LiDAR range image is trained using batch size 96 and learning rate 4e-4 for 200 epochs. The discriminator takes effect after 1000 iterations. In the second stage, we train the LiDAR LDM from scratch using batch size 96 and learning rate 1e-4 for 2000 epochs. The model includes the text prompt and 3D range-view bounding box condition modules with drop-rate 0.25 for either condition during training. In the last stage, we train the entire framework in an end-to-end manner with module trainable states illustrated in Fig. 2 using condition drop-rate 0.25 as well. The entire model is trained for 250 epochs with learning rate 8e-5 and batch size 24 in our main experiments. For ablation studies, we reduce the epoch number to 80 for efficiency.

##### Evaluation metrics.

In our quantitative evaluation, we apply the FID metric from MagicDrive (Gao et al., ), VSC metric from BEVGen (Swerdlow et al., ), MMD and JSD metrics from LiDARGen (Zyrianov et al., ). For our proposed DAS metric, we run pretrained DepthAnythingV2 model (Yang et al., ) with ViT-B backbone (Dosovitskiy, ) on the synthetic images to estimate the disparity (Yang et al., ). Then, we project the synthetic point clouds to multi-view images to get sparse disparity. We normalize the projected sparse disparity to the same scale as the estimated disparity. The mean absolute error is fetched as our DAS metric.

## Appendix B Limitation and Future Works

This paper concentrates on the joint generation of consistent multi-modality data. In this case, due to the limitation of accessible computational resources, we do not include some natural extensions of our X-Drive framework. Firstly, our method is limited to the generation of single-frame point clouds and multi-view images. In fact, it is intuitive to combine existing temporal attention modules like Wen et al. () with our X-Drive. Secondly, we can also incorporate some extra conditions for data synthesis such as HD maps (Gao et al., ) into our framework for more controllable generation. Lastly, due to the distribution of nuScenes training set (Caesar et al., ), we cannot synthesize some specific scenarios such as snowy weathers. We will try to extend our framework to deal with above limitations in our future work.

## Appendix C Qualitative Results

In this section, we demonstrate additional examples of our synthetic multi-modality data in Fig. 8. X-Drive can generate realistic point clouds and multi-view images with cross-modality consistency based on the input conditions.

Figure: Figure 8: Additional multi-modality data synthesis results using our proposed X-Drive.
Refer to caption: x8.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Is there a small mistake in Eq. (5)?
soundness: 3
strengths: + This paper introduces the first framework for the joint generation of LiDAR and camera data for autonomous driving scenarios. 

+ The proposed cross-modality epipolar condition module to enhance modality consistency makes sense and works well. 

+ The paper is well-written and easy to follow.
summary: This paper introduces X-DRIVE, a novel framework for generating consistent multimodal data, specifically LiDAR point clouds and multi-view images, for autonomous driving scenarios. The key innovation of this paper is a cross-modality epipoplar condition module that bridges the geometrical gap under spatial ambiguity between point clouds and multi-view images, significantly enhancing modality consistency. Experimental results have demonstrated the effectiveness of the proposed method.
weaknesses: 1. This paper presents a well-designed diffusion architecture for joint LiDAR and image data generation. My major concern is the novelty. This paper's main contribution is a cross-modality epipolar condition module for consistency constraints. However, epipolar attention for multi-view consistency has been extensively explored in multi-view diffusion models [1, 2, 3]. 

[1] Liu, Yuan, et al. "Syncdreamer: Generating multiview-consistent images from a single-view image." arXiv preprint arXiv:2309.03453 (2023).
[2] Shi, Yichun, et al. "Mvdream: Multi-view diffusion for 3d generation." arXiv preprint arXiv:2308.16512 (2023).
[3] Yang, Jiayu, et al. "Consistnet: Enforcing 3d consistency for multi-view images diffusion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

2. Experiments are only conducted on one dataset, nuScenes. 

3. It would be great to detail what kind of and how many GPUs are used, as well as the training/inference time.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Q1: Baseline setting for Tab 1: I don’t completely understand how MagicDrive + RangeLDM was combined to obtain cross-modality score? Was there anything done to ensure alignment between the different modalities from the two models?
- Q2:  I was wondering if the authors made the decision of studying this problem in range-image space based on empirical results or out of convenience (prior work)? In other words, was denoising in pointcloud space studied for this problem?
- Q3: (suggestion) I wonder if the evaluation metrics are unable to capture objectness of pointclouds. Athough not typical in generative learning, techniques like Iterative closest point (ICP) could be useful in measuring shape discrepancies.
- Q4: Performing denoising in each modality while encoding information from the other modality through the denoising time steps seems expensive. It would be helpful to provide the computation time split between the different stages? If possible, also relate it to time taken for unimodal generations.
- Q5: Was the zero initialization from ControlNet deemed necessary empirically? I’m curious if the authors observed any forgetting without it.
- Q6: It looks like the results for lidar to camera is far better than the multimodal and camera to lidar results to me. Do the authors agree? If so, since generating point clouds is difficult, would disnetangling the joint process to a sequential approach of first generating camera data and then performing conditional lidar generation be feasible?
- Q7: Have the authors looked at GenMM [6]? While they do not specifically tackle multi-modal generations, they target alignment between the two modalities using in-painting diffusion models. I wonder on similar lines to Q5 if their approach can be conducive to a sequential approach of using reference patches from camera generations to condition generating point cloud data leveraging in-painting.

Prioritization order: Q1 > Q2 = Q4 > Q7 = Q6 > Q5 > Q3

I welcome the authors to correct me if there was any misunderstanding on my part.
soundness: 3
strengths: - S1: The work addresses an important problem of lacking aligned lidar camera data. Solutions towards this can reduce/elimintate post-processing work of performing motion compensation via scene flow between consecutive pointcloud frames to address the lack of alignment. This is a cumbersome process.
- S2: The epipolar line based matching technique is neat, intuitive, and has been successfully been used before in literature [1].
- S3: The manuscript is well written and easy to follow. I also like the multi-view consistency results.

[1] Consistent View Synthesis with Pose-Guided Diffusion Models - https://arxiv.org/pdf/2303.17598
summary: - This work proposes a novel technique to perform driving based lidar data generation in the form of range images, and multi-camera data generation while ensuring cross-modal alignment by leveraging latent diffusion models.
- Prior works address multi-camera image generation and lidar based generation individually while this work aims to perform joint generation of lidar and camera data. The major contribution here is in aligning the modalities during generation.
- During denoising of a first modality (camera or lidar), the information from the other modality (lidar or camera) is encoded and transformed into the geometrical space of the first modality (camera or lidar) to ensure alignment. They do this by finding matches from a second modality for a query from first modality by traversing the epipolar line in the second modality corresponding to the query.
- They show qualitative and quantitative results for conditional and unconditional generations. In addition, the authors also display controllability via bounding boxes and language interfaces.
weaknesses: - W1: Mapping pointclouds to range images leads to information loss. It is understandable the authors use range images since most of the existing lidar generation work uses range images and in addition, image based generation techniques are much more mature than pointcloud counterparts. However, the major goal here is to ensure alignment between the modalities and not to perform unimodal generations which have been addressed before. Therefore, I would expect point clouds to be effective in filling gaps caused by information loss especially during occlusions and in far ranges.
- W2: I find the qualitative results of specifically the lidar generations lacking when viewed in the form of point clouds. Since the application of this work is to be used for training models, having distributional similarities (elaborated below) with real points is important to serve as valuable indictive biases and I doubt its utility without it.
    - Pattern realism: The lidar patterns shown in the pointcloud figures do not look realistic to me. For instance, the ground points on a rotating lidar typically show-up as thick concentric circles whereas results in supp. Fig 8 look very artificial to me. Comparing this to real pointclouds or even other synthetic lidar generation techniques from literature such as [2, 3], I see a gap that might be important to ignore for a downstream model.
    - Object realism: The point clouds also lack object shape or object realism in the figures presented in Fig 6 of main paper. For instance, vehicles have a typical shape based on their orientation which are useful biases for a downstream network that consumes this data. These patterns are very evident in other synthetic techniques [2, 3] but seems to be lacking here. Similarly, on the range images for the multi-modal generations in Fig 4, very prominent cars in the scene do not have as many lidar points as I would expect. This was an objective of the work as stated in the manuscript (L64) which I don’t believe has been fully fulfiled.
    - Following similar theme, the points on the pedestrian in Fig 4 do not seem realistic to me.
- W3: Related to W2, since the larger goal of this work is to generate data that can be used for training, I would have expected to see results showcasing this utility. The manuscript in the current form does not convey the effectiveness of the data towards this.
- W4: The authors state one of the application is to address long-tail scenarios (L52). They do show results in rain and bounding box can be a way to inject situtations such as cut-ins. However, these are mainly shown in the visual space currently which is also illustrated by prior works such as [4, 5]. The primary evaluation of the work proposed here is not in judging effectiveness of visual generation but alignment between cross-modal generations. The current results in the manuscript do not show such OOD scenarios which can be validated in the point cloud and image space.

I am weighting W2 > W1 > W3 = W4.

Overall, I like the work and the problem being tackled. However, I do not believe the results are quite there yet to be useful. Therefore, I am inclined to reject this work but I am open to changing my opinions based on the rebuttal and reviewer discussion.

Minor comments:

- Typo in L259 - Fourier
- L294 - I think you mean split vertically?
- Typo in L424.


References:

[2] Towards Realistic Scene Generation with LiDAR Diffusion Models - https://arxiv.org/pdf/2404.00815

[3] RangeLDM - https://arxiv.org/pdf/2403.10094

[4] MagicDrive - https://arxiv.org/pdf/2310.02601

[5] BEVGen - https://arxiv.org/pdf/2301.04634

[6] GenMM - https://arxiv.org/pdf/2406.10722

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - In Figure 1(b), what does the distribution at the bottom represent?
- In Figure 2, how is the generation of the LiDAR point cloud achieved?
- What are the additional time and memory costs of the designed module?
- What is the average generation time for one experiment?
soundness: 2
strengths: - The focus on consistency between LiDAR and image generation provides insights to this field.
summary: This work proposes X-Drive, explicitly addressing LiDAR-image consistency in driving data synthesis. It designs (1) mutual conditioning between corresponding local regions across modalities, (2) an epipolar line-based conditioning module, and (3) multi-source input conditions to mitigate generation inconsistencies. Experiments on nuScenes validate the efficacy of the proposed method.
weaknesses: - Lack of quantitative comparison with baselines. In Table 1, the number of experiments conducted is insufficient to fully support this paper. The authors are encouraged to compare with more previous methods and implement additional vanilla baselines to better illustrate the advantages of X-Drive. Table 1 could be divided by modality and enriched separately.
- Lack of ablation studies. In Table 3, the ablation analysis is presented at a very coarse level. The authors are encouraged to detail the design of each module and analyze their individual impact on the final results.
- Marginal improvements. In Table 1, for JSD in the C→L setting, X-Drive only surpasses MagicDrive by 0.002, while RangeLDM surpasses LiDARGen by 0.106. For FID in the L→C setting, X-Drive only surpasses MagicDrive by 0.19, whereas MagicDrive surpasses BEVGen by 8.65.
- Numerous typos. For example, "several works emerges" in L103, "We extend the vanilla diffusion models to modeling" in L186-187, and "Our FID metric is outoerformed by MagicDrive" in L423-424. These errors should be reviewed and corrected to enhance the clarity and readability of the paper.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: While the work presents valuable theoretical contributions, significant improvements in code availability, documentation, and practical validation are needed for broader impact and reproducibility.

### 1. Code Release Improvements
- Release code on popular platforms (GitHub/GitLab)
- Provide comprehensive documentation
- Include example applications
- Add benchmark results reproduction scripts
soundness: 3
strengths: - First work to achieve joint generation of LiDAR point clouds and multi-view camera images with cross-modal consistency
- Novel epipolar-based cross-modal condition module addressing alignment between point clouds and images in different geometric spaces
- Multi-level controllable generation incorporating text, bounding boxes, images and point clouds as conditions
summary: # Review Analysis of X-DRIVE Paper

## 1. Methodology and Key Innovations

### Primary Methods
- Proposed X-DRIVE framework: A dual-branch latent diffusion model architecture for simultaneous generation of LiDAR point clouds and multi-view camera images
- Designed Cross-modality Epipolar Condition Module to enhance consistency between different modalities
- Introduced 3D bounding boxes for geometric layout control and text prompts for attribute control (e.g. weather and lighting)

### Key Innovations
- First work to achieve joint generation of LiDAR point clouds and multi-view camera images with cross-modal consistency
- Novel epipolar-based cross-modal condition module addressing alignment between point clouds and images in different geometric spaces
- Multi-level controllable generation incorporating text, bounding boxes, images and point clouds as conditions

## 2. Innovation Analysis

### Limitations in Innovation
- Weak theoretical foundation for cross-modal consistency, lacking rigorous proof of epipolar-based design effectiveness
- Incomplete solution for "spatial ambiguity", relying solely on epipolar geometry may not fully resolve 3D space ambiguities
- Limited innovation in generation control, mainly applying existing control methods to new scenarios

## 3. Experimental Analysis

### Required Additional Experiments
- Evaluation of generated data utility in downstream tasks (3D object detection, trajectory prediction)
- Need to adopt more advanced transformer architectures like DiT instead of U-Net
- Consider replacing CLIP with T5 for potentially better performance
- Address low resolution issues affecting geometric shapes of distant objects (The current resolution is too low. Objects in the distance have no geometric shapes at all. However, the point cloud may have relatively complete and accurate geometric shapes in the distance. A larger resolution should be used, and because the generated image has slightly less geometric shapes in the distance, the effect is not good, simple and practical super-resolution model. It can only increase the resolution but cannot improve the distant geometry.)
- Investigate video generation capabilities with 3D VAE and 3D attention mechanisms
- Develop quantitative metrics for cross-modal consistency evaluation

## 4. Ablation Study Analysis

### Missing Ablation Studies
- Detailed component-wise analysis of cross-modal epipolar condition module
- Impact assessment of different condition types on generation quality
- Sensitivity analysis of key hyperparameters
- Investigation of different training strategies

## 5. Advanced Experiments (Optional)

### Long-tail Data Generation
- Potential for generating rare/corner cases
- Ability to synthesize challenging scenarios
- Evaluation of model performance on long-tail distributions

## Overall Assessment
While the paper presents novel contributions in cross-modal data generation, it requires strengthening in theoretical foundations, experimental validation, and analysis depth. The framework shows promise but needs additional experiments and rigorous evaluation to fully demonstrate its capabilities.
weaknesses: ## Limitations in Innovation
- Weak theoretical foundation for cross-modal consistency, lacking rigorous proof of epipolar-based design effectiveness
- Incomplete solution for "spatial ambiguity", relying solely on epipolar geometry may not fully resolve 3D space ambiguities
- Limited innovation in generation control, mainly applying existing control methods to new scenarios

## Experimental Analysis

### Required Additional Experiments
- Evaluation of generated data utility in downstream tasks (3D object detection, trajectory prediction)
- Need to adopt more advanced transformer architectures like DiT instead of U-Net
- Consider replacing CLIP with T5 for potentially better performance
- Address low resolution issues affecting geometric shapes of distant objects (The current resolution is too low. Objects in the distance have no geometric shapes at all. However, the point cloud may have relatively complete and accurate geometric shapes in the distance. A larger resolution should be used, and because the generated image has slightly less geometric shapes in the distance, the effect is not good, simple and practical super-resolution model. It can only increase the resolution but cannot improve the distant geometry.)
- Investigate video generation capabilities with 3D VAE and 3D attention mechanisms
- Develop quantitative metrics for cross-modal consistency evaluation

## Ablation Study Analysis

### Missing Ablation Studies
- Detailed component-wise analysis of cross-modal epipolar condition module
- Impact assessment of different condition types on generation quality
- Sensitivity analysis of key hyperparameters
- Investigation of different training strategies

Meta-review: The paper proposes a method to synthesize consistent lidar and camera images for outdoor driving scenario using diffusion models. While it bares similarities to methods in single modality generation, the authors use an additional branch where the consistency between the two are enforced through the use of epipolar constraint/module. It also claims to be the first cross-modality generation method. We have read through the referee reports and the author responses. There are several common threads of concern involving epipolar constraint, realistic generation, and experiments (ablations, downstream applications, etc.). While some of the reviewers did not respond or confirm whether their questions or concerns were addressed, based on the author responses, we feel that they were mostly addressed. That said, the common concerns shared across reviewers included critical points, especially on experiments, and we recommend that revise the manuscript according to the reviewers' feedback.

Decision: Accept (Poster)