LiDAR Data Synthesis with
Denoising Diffusion Probabilistic Models
Kazuto Nakashima and Ryo Kurazume (Kyushu University)
TL;DR: A diffusion-based generative model for LiDAR generation and completion

1Motivation

Deep generative models as 𝑝 𝑥§ LiDAR point clouds are crucial for autonomous mobile robots
to enable their perception (e.g., semantic segmentation)§ The sparsity of point clouds, such as due to adverse weather orlimited laser beams, poses a challenge to the performance§ How can we restore the degraded LiDAR point clouds?
𝒚 (degraded)2

Related Work𝒙Method

Approach Range Reflectance Format Dataset

LiDAR-VAE/GAN [1]

VAE/GAN

Image KITTI

DUSty v1/v2

[2,3]

GAN

Image KITTILiDARGen[4]

SGM

Image KITTI-360, nuScenes

DDPM

Image KITTI-360, KITTIR2DM (ours)§ Both the range and reflectance are indispensable for perception

✗✗

✗

𝑝 𝒙𝒚

✓

§ LiDARGen [4] is most closely related to ours
Perception§ Trained LiDAR data w/ discrete-time diffusion model (SGM)

Approach: Build a data prior 𝑝 𝒙 and approximate 𝑝 𝒙 𝒚§ The sampling w/ fixed >1k steps is slow

R2DM (Range–Reflectance Diffusion Model)§ Data representation (diffusion space):Signal𝒙𝒛" 𝑞 𝒛! 𝒙

= 𝒩 𝛼! 𝒙, 𝜎!#𝑰𝒛!

Latent𝒛~𝒩 𝟎, 𝐈3§ Equirectangular images by spherical projection§ Log-scale depth [4]: log 𝒙 + 1 / log 𝑥!"# + 1
improves the geometric resolution of nearby points

Spherical
projection§ Noise prediction NN 𝝐) (⋅) is Efficient U-Net [6] w/

Forward diffusion
𝑞 𝒛! 𝒛"§ horizontally-circulated kernel operation (e.g. conv)

NN 𝝐) ⋅Inverseprojection§ spatial inductive bias using a positional encoding

LiDAR point cloudTimestep 0
SNR= 𝛼!#/𝜎!# ∞𝑠𝑡§ Continuous-time DDPM [5]10Noised sample 𝒛! ~𝑞 𝒛! |𝒙§ Training loss: ℒ = 𝔼𝒙,%~𝒰 /,0 ,𝝐~𝒩 /,3Beam angles

Reverse diffusion𝑝 𝒛" 𝒛!

Noise prediction𝝐 − 𝝐) 𝛼% 𝒙 + 𝜎% 𝝐, 𝜆%Infinite steps Additive noise

SNR schedule

44log SNR

§ Sampling: recursively evaluate 𝑝 𝒛$ |𝒛% at discretized timesteps
parameterized with the given 𝒛!
and 𝝐) estimated by the NN§ NFE (number of sampling steps) vs Scores:
MMD (BEV) ↓NN 𝝐) ⋅log SNR 𝜆!

Spatial bias 𝒛!

Noise 𝝐𝒛"

§ Post-hoc completion based on RePaint [7]§ After each 𝑝 𝒛$ |𝒛% , overwrite the known region 𝒎*'&'()'
𝒛$ = 𝒎⨀𝒛&'()'+1−𝒎⨀𝒛$$Mask of 𝒚(𝒚 = 𝒎𝒙)

Noised known pixels~𝑞 𝒛" |𝒚Denoised unknown pixels~𝑝 𝒛" |𝒛!

How accurate is the completion 𝑝 𝒙 𝒚 ?

4Evaluate a distributional similarity at the image/point/BEV levels
between generated samples and real samples of datasets⋯Fourier features [6]The number can be determinedaccording to the computational trade-off

Unconditional Generation

FPD (point cloud) ↓Positional
Encoding𝜃, 𝜙How faithful and fast is the generation 𝑝(𝒙) ?

FRD (image) ↓Efficient U-Net

5

Beam-level Upsampling
Upsample the LiDAR images with 4x fewer beams and evaluate
pixelwise precision (MAE) and semantic consistency (IoU) to GTs
Input(16×1024)

Bicubic
interp.

ILN [8](supervised)

LiDARGen [4](diffusion)

R2DM(ours)

MAE (range) ↓:MAE (reflectance) ↓:2.297
0.108

2.237
N/A

1.551
0.080

0.923
0.050

18.54

21.80

22.46

34.44

GT(64×1024)

JSD (BEV) ↓LiDARGenR2DM

Time per step: R2DM (15.77 ms, 31M params), LiDARGen (47.17 ms, 30M params)
Predicted byRangeNet-53 [9]KITTI-360§ Generated examples:

IoU (%) ↑:§ More examples:
Baseline (LiDARGen [4])R2DM (ours)

Real data

Baseline (DUSty v2 [3])R2DM (ours)

16/64 beams

8/64 beams

10% points

No road(reflectance only)

KITTI Raw

Real data

16/64 beams

No range

References: [1] Caccia et al. IROS’19, [2] Nakashima et al. IROS’21, [3] Nakashima et al. WACV’23, [4] Zyrianov et al. ECCV’22, [5] Kingma et al. NeurIPS’21, [6] Saharia et al. NeurIPS’22, [7] Lugmayr et al. CVPR’22, [8] Kwon et al. ICRA’22, [9] Milioto et al. IROS’19