LiDAR Data Synthesis withDenoising Diffusion Probabilistic Models
Kazuto Nakashima

Ryo Kurazume

Faculty of Information Science and Electrical Engineering
Kyushu University, JapanMotivation§ LiDAR point clouds are crucial for autonomous robots to enable perception(e.g., semantic segmentation)§ Sparsity of the point clouds poses a challenge to the performance(e.g., adverse weather and limited beams)𝒚 (degraded)

✗✗

𝒙

✗

𝑝 𝒙𝒚

✓

Approach: Build a data prior 𝑝 𝒙 and approximate 𝑝 𝒙 𝒚Perception

Related work𝑝 𝒙 : Generative modeling of LiDAR dataMethodApproachFormatLiDAR-VAE[Caccia et al. IROS’19]

VAE

Image

LiDAR-GAN[Caccia et al. IROS’19]

GAN

Image

DUSty

[Nakashima et al. IROS’21]

GAN

Image

DUSty v2

[Nakashima et al. WACV’23]

GAN

Image

LiDARGen[Zyrianov et al. ECCV’22]

Diffusion

Image

RangeReflectance§ LiDARGen trains a discrete-time diffusion model on the image representation§ Sampling with fixed >1k steps is slow (12 mins/batch)§ The sample fidelity still has a gap compared to real dataOur approachOur approach

R2DM: Range–Reflectance Diffusion ModelForward diffusion
Spherical
projection

Continuous-time DDPM
Reverse diffusion

LiDAR point cloud
Range image(+ reflectance)

Latent variable

R2DM: Range–Reflectance Diffusion Model

Data representation
Following LiDARGen [Zyrianov et al. ECCV’22] :§ Spherical projection§ Log-scale representation

Metric depth
Spherical
projection
log 𝑑 + 1
log 𝑑!"# + 1

Gain the geometricresolution of nearby points

LiDAR point cloud
Range image(+ reflectance)

Log-scale depth

R2DM: Range–Reflectance Diffusion Model

Generative modelingTraining the continuous-time DDPM [Kingma et al. NeurIPS’21] on the image representation
Forward diffusion

Continuous-time DDPM

Data space

Reverse diffusion
U-Net

Latent space

Trained to predict noise✓ The number of steps is adjustable in sampling

R2DM: Range–Reflectance Diffusion Model

Noise prediction model (U-Net)

Latent variable 𝒛$

𝝐-prediction
U-Net

𝒛$ = 𝛼$ 𝒙 + 𝜎$ 𝝐

R2DM: Range–Reflectance Diffusion Model

Noise prediction model (U-Net)§ Efficient U-Net [Saharia et al. NeurIPS’22]
Latent variable 𝒛$

𝝐-prediction

ResBlock

ResBlock

Down
ResBlock

UpResBlock

Down
ResBlock

UpResBlockDownResBlock

SelfAttn

UpResBlock

SelfAttn

Our modifications:§ Range & reflectanceequirectangular image§ All kernel operations are
circulated horizontally§ Improved spatial bias
for geometric fidelity

R2DM: Range–Reflectance Diffusion Model

Spatial inductive bias
§ The generated pixels must be accurate and also spatially well-alignede.g.

Original range image

1.1x zoom

Distorted

R2DM: Range–Reflectance Diffusion Model

Spatial inductive bias
§ LiDARGen [Zyrianov et al. ECCV’22] proposed to use azimuth & elevation beam angles
as additional channels of the U-Net input like CoordConv [Liu et al. NeurIPS’18]
Latent variable 𝒛$
𝝐-prediction
Beam angles 𝜃, 𝜙

U-Net

Horizontal discontinuity & low spatial frequency (< 1)R2DM: Range–Reflectance Diffusion Model

Spatial inductive bias
Our approach:
Mapping to the wideband & cylindrical spatial representation

Latent variable 𝒛$
Beam angles 𝜃, 𝜙Positional
Encoding

Spatial bias
U-Net

Fourier features[Tancik et al. NeurIPS’20]⋮

R2DM: Range–Reflectance Diffusion Model

Unconditional generation

How can we control the generation 𝑝 𝑥 with the degraded 𝑦?

R2DM: Range–Reflectance Diffusion Model

Completion by conditional generation
We adopt RePaint (an inpainting technique using DDPMs) [Lugmayr et al. CVPR’22]
𝑡=1
Unconditional
𝒙~𝑝 𝒙
Noise

Denoising(prediction)

Denoising(prediction)

𝑡=0

ResultR2DM: Range–Reflectance Diffusion Model

Completion by conditional generation
We adopt RePaint (an inpainting technique using DDPMs) [Lugmayr et al. CVPR’22]
Cycle the step to harmonize the mixed regions
𝑡=1
Conditional𝒙~𝑝 𝒙 𝒚Denoising(prediction)

Denoising(prediction)

Noise𝑡=0Result
Overwrite the known regions
Noising(analytic)

𝑡=0Known regions
Given 𝑦R2DM: Range–Reflectance Diffusion Model

Completion by conditional generation
8/64 beams

Input

Output

10% random points

Input

OutputNo roadInput

OutputEvaluation
Generation & Completion

Evaluation

How faithful is the generation 𝑝 𝒙 ?
10k samples
Random sampling𝑧 ∼ 𝒩 0, 𝐼

Distributional similarity
Real samples from the LiDAR dataset(KITTI-360)

Evaluation

Results
Real

LiDARGen[Liao et al. TPAMI’22][Zyrianov et al. ECCV’22]

Quality

Image-level (FRD ↓)Point-level (FPD ↓)579.3990.29

R2DM (ours)

153.743.92

Evaluation

How accurate is the completion 𝑝 𝒙 𝒚 ?
Task: Beam-level upsamplingSparse input(16×1024)

Upsampling

Full resolution(64×1024)&Semantic
segmentation

MAE

mIoU

(mean absolute error)(intersection-over-unions, %)Evaluation

Results
Bicubic

ILN

LiDARGenR2DM (ours)[Kwon et al. ICRA‘22] [Zyrianov et al. ECCV‘22]

MAE:

2.297

2.237

1.551

0.923

mIoU (%):

18.54

21.80

22.46

34.44

GT

Evaluation

Sampling efficiency

In generating a single sample:

LiDARGen#Params

Time [ms]

29.7M

47.17

w/ RefineNet
U-Net

U-Net

U-Net

3x faster

U-Net

R2DM
Execution time per step

31.1M

w/ modified Efficient U-Net

15.77

Evaluation

Sampling efficiency

In generating a single sample:
U-Net

U-Net

U-Net

U-Net

Score (FRD)

worse

LiDARGen(1,160 steps)

R2DM

better# steps vs fidelity tradeoff

Number of steps
betterworse

R2DM reaches LiDARGen in 32 steps(1,160 steps)

Conclusion§ R2DM, a continuous-time DDPM for LiDAR data generation & completion§ Demonstrated unconditional generation & beam-level upsampling tasks§ Pre-trained models & demos are available atkazuto1011/r2dm§ More details & ablation study are on our paper/poster!