Generative Range Imaging for Learning
Scene Priors of 3D LiDAR Data
Kazuto Nakashima*

Yumi Iwashita

Ryo Kurazume

Kyushu University

NASA/Caltech JPL

Kyushu UniversityMotivation
3D LiDAR sensors are important for robotics applications

Obstacle detection[Lang et al. CVPR’19]

Semantic segmentation[Behley et al. ICCV’19]

Motivation
Issue: domain gaps in LiDAR perception tasks

High resolution
DenseLow resolution
SparseRay-drop(missing points)

Simulation
CleanRealNoisy

We propose LiDAR data priors for bridging the domain gaps
using deep generative models
images from [Yi et al. CVPR’21]

Towards LiDAR Generative Models
Range 𝑥!

Range image

Range image
Latent space

Training GANs on LiDAR images[Caccia et al. IROS’19]

DatasetSamples

Latentspace

Probability 𝑥"

Noise-aware GAN on LiDAR images[Nakashima et al. IROS’21]+ Robustness on ray-drop noises
Training GANs w/ invertible corruption[Bora et al. ICLR’18] [Kaneko et al. CVPR’20]– Fixed resolution = LiDAR dependence– No demonstration on perception tasks

LiDAR Range Images as 2D Neural Fields
Control 𝐺 weights

Range 𝑥!

Elevation 𝜙

Scene prior 𝑧
QueryRange image

𝜃, 𝜙𝐺

Azimuth 𝜃Generator(Neural Fields)

Probability 𝑥"

𝑀𝐷

Differentiable
ray-drop rendering

Discriminator(CNN)[Nakashima et al. IROS’21]

LiDAR Range Images as 2D Neural Fields
Control 𝐺 weights

Range 𝑥!

Elevation 𝜙

Scene prior 𝑧
QueryRange image

𝜃, 𝜙

𝐺

Azimuth 𝜃Generator(Neural Fields)

Range 𝑥!

𝐷Differentiable
ray-drop rendering

Discriminator(CNN)[Nakashima et al. IROS’21]

Angles 𝜃, 𝜙LiDAR

Scene

Probability 𝑥"

𝑀Surface 𝐺 𝑧

ReturnedMissing

Analogy of LiDAR measurement

LiDAR Range Images as 2D Neural Fields
Control 𝐺 weights

Range 𝑥!

Elevation 𝜙

Scene prior 𝑧
QueryRange image

𝜃, 𝜙𝐺Azimuth 𝜃Generator(Neural Fields)

𝑀

𝐷Differentiable
ray-drop rendering

Discriminator(CNN)

Probability 𝑥"[Nakashima et al. IROS’21]

𝑥"𝑚~Bernoulli 𝑥"(non-differentiable)𝑚𝑥!
⨀Gumbel-Sigmoid(continuous relaxation)

LiDAR Range Images as 2D Neural Fields
Control 𝐺 weights

Range 𝑥!

Elevation 𝜙

Scene prior 𝑧
QueryRange image

𝜃, 𝜙𝐺Azimuth 𝜃Generator(Neural Fields)

Probability 𝑥"𝑀𝐷

Differentiable
ray-drop rendering

Discriminator(CNN)[Nakashima et al. IROS’21]

Training data (KITTI)

Complete depth

Ray-drop probability

Latent interpolation w/ our GAN

Fidelity and Diversity
Image-level:
SWD on inverse depth maps

l-WGAN

Ours

Point-level:

Feature-level:

JSD, COV, MMD, and 1-NNA

FPD and MMD2 on PointNet features

RealApplications

Decomposition
TargetOptimize latent code
Reconstruction

Complete depth

Ray-drop probability

Restoration
TargetOptimize latent code
Reconstruction

Complete depth

Ray-drop probability

UpsamplingTarget

1× reconstruction

1. Optimize latent code

2× reconstruction

2. Densify angular queries

4× reconstruction

Sim2Real Sementic Segmentation
Training:

Automatic & HQ
Segmentation
model

Simulator

LiDAR

LiDAR simulation

Test:

Real

LiDAR

LabelsSim2Real Sementic Segmentation
Training:

Unrealistic
Segmentation
model

Simulator

LiDAR

Labels

Test:

Degraded
Segmentation
model
Real

LiDAR

GT

Sim2Real Sementic Segmentation
Training:

Unrealistic

Realistic
Rendering ray-dropby GAN inversion

Simulator

LiDAR

Segmentation
model
LiDAR

Labels

LatentTest:

Adapted
Segmentation
modelReal

LiDAR

GT

Sim2Real Sementic Segmentation
InputBaseline

Ours

1.7% mIoU

46.3% mIoU

w/ our ray-drop rendering

GTSim2Real Sementic Segmentation
GTA-LiDAR (simulation) → KITTI (real) [Wu+, ICRA’19]

Thank you!
Code and models are available athttps://kazuto1011.github.io/dusty-gan-v2