Learning to Drop Points for LiDAR Scan Synthesis
Kazuto Nakashima and Ryo Kurazume

I. I NTRODUCTION
3D scene understanding is indispensable for mobile robots
to detect obstacle objects, ﬁnd traversable paths, and explore
the real world. A typical representation of 3D scenes isa point cloud, which can be measured by various rangescanning devices such as 3D LiDARs and RGB-D cameras.
In particular, 3D LiDARs are widely used for autonomous
driving systems to enable their perception capabilities such
as SLAM, object detection, and semantic segmentation.
A 3D LiDAR calculates distances based on the time-ofﬂight of pulsed laser emitted and reﬂected at multiple elevation/azimuth angles. However, to gain the angular resolution
and precision, it requires many scanner units and sufﬁcienthousing size, which could conﬂict with system requirements.
Moreover, the measured points could have noises underadverse weather. The purpose of this study is to reconstructhigh-quality point clouds from such limited or corrupted
observations. However, it is non-trivial to manipulate a number of raw points with semantic consistency. One promising
approach is to learn a generative model of point clouds as
scene priors and to ﬁnd the high-quality sample nearest to
the observation.
Generative models have received considerable attention
owing to recent advances in representation learning withdeep neural networks. In particular, generative adversarial
Kazuto Nakashima and Ryo Kurazume are with Faculty of Information Science and Electrical Engineering, Kyushu University, 744 Motoka
Fukuoka, Japan. k nakashima@irvs.ait.kyushu-u.ac.jp,

kurazume@ait.kyushu-u.ac.jp

2D generation

bilityra
Measu
Latent space

Inversedepth

Differentiablesamplingn
AR scatic LiDps
Syntheoint drophitw

2D mapSpherical
Cartesian(a) Decomposed synthesis of LiDAR scans

Point clouds

Abstract— 3D laser scanning by LiDAR sensors plays an important role for mobile robots to understand their surroundings.
Nevertheless, not all systems have high resolution and accuracy
due to hardware limitations, weather conditions, and so on.
Generative modeling of LiDAR data as scene priors is one of thepromising solutions to compensate for unreliable or incomplete
observations. In this paper, we propose a novel generativemodel for learning LiDAR data based on generative adversarial
networks. As in the related studies, we process LiDAR data as
a compact yet lossless representation, a cylindrical depth map.
However, despite the smoothness of real-world objects, many
points on the depth map are dropped out through the laser
measurement, which causes learning difﬁculty on generativemodels. To circumvent this issue, we introduce measurement
uncertainty into the generation process, which allows the model
to learn a disentangled representation of the underlying shapeand the dropout noises from a collection of real LiDAR data.
To simulate the lossy measurement, we adopt a differentiablesampling framework to drop points based on the learned
uncertainty. We demonstrate the effectiveness of our method
on synthesis and reconstruction tasks using two datasets. We
further showcase potential applications by restoring LiDAR
data with various types of corruption.

Target (unseen real data)

Reconstruction(b) Reconstruction of unseen data from KITTI [1] test set
Fig. 1. A concept of our proposed generative model for LiDAR data. (a)With only noisy training data, our proposed method can learn an underlying
complete depth map and a measurability map that simulates dropout noises.
(b) The trained model can be used to reconstruct unseen data.

networks (GANs) [2] are a popular framework for building
a generative model. GANs have been actively studied on 2D
tasks such as generating photo-realistic natural images [3, 4],while they can also be used to model 3D data such as point
clouds. Several studies [5, 6] proposed generative models that
produce point clouds as a set of unordered points and demonstrated on data uniformly sampled from CAD models [7] In
contrast, the point clouds by 3D LiDARs are often large,
non-uniform, and sparse, as they are measured by radiatedlasers centering on the sensor head. In other perception tasks
with LiDAR data, projection-based 2D representations havealso been studied, such as sensor-view depth images byspherical projection [8, 9] and bird’s eye view [10]. The 2D
approaches have advantages in computational efﬁciency: theprocessing can be designed with 2D convolutional networks,
less problem on the sparsity, and the representation is compact yet lossless for spherical projection. In semantic point
segmentation [9, 10] and object detection tasks [8], theseapproaches have gained performance in practice.
The goal of this study is to build a generative modelof LiDAR data based on the 2D representation [8]–[12].
In particular, we use the sensor-view representation. Caccia
et al. [11] is closely related to our task, which trained
GANs and variational autoencoders (VAEs) with Cartesianpoints projected onto sensor-view grids. However, we foundthat training generative models on this 2D representation

is still challenging because of LiDAR-speciﬁc noises. As
seen in Fig. 1(b), the projected image involves scattereddropout noises. The noises are due to the reﬂection failureof laser measurement, hereinafter called point-drops. Caccia
et al. [11] handled this issue by interpolating all the pointdrops by neighbor pixels, however, the heuristic imputation
could cause unnatural structures.
In this paper, we introduce a novel GAN framework
to synthesize depth with uncertainty from styles, named
DUSty. Our key idea is to introduce measurement uncertainty into the generation process so that the underlying
complete signals can be modeled implicitly. As shown in
Fig. 1(a), our model produces a complete depth map and thecorresponding measurability to sample realistic dusty pointdrops. Our proposed model can be trained only with raw
LiDAR data involving point-drops. We evaluated our method
on two LiDAR datasets for driving scenes: KITTI [1] andMPO [13]. We demonstrate that our method successively
synthesizes realistic LiDAR data, learning the underlying
complete shape. We also demonstrate the reconstruction
capability for unseen real data in various corruption settings.
The code will be available at https://github.com/
kazuto1011/dusty-gan.
The main contributions can be summarized as follows:• We propose a noise-aware GAN framework DUSty,
which models a complete depth map and the corresponding measurability from real LiDAR data.
• We introduce a differentiable relaxation to learn thediscrete distribution of point-drops.
• We demonstrate the effectiveness of our approachon synthesis and reconstruction tasks on two LiDAR
datasets.
II. R ELATED W ORK
A. Synthesizing LiDAR Data
Generative modeling of point clouds is a challenging
task emerged in recent years. Most studies focused on
small point sets sampled from CAD objects [5], but rarely
on LiDAR point clouds. There have been studies basedon simulated environments [9, 12], but their diversity was
limited to deﬁned cases. Only Caccia et al. [11] worked
on modeling LiDAR data with two popular frameworks fordeep generative models: variational autoencoders (VAEs) and
GANs. The authors revealed that 2D representation wasmuch better for LiDAR data than for point sets. However, thestudy evaluated quantitative performance on reconstruction
tasks only for VAEs. In this paper, we provide the results ofboth quality and diversity metrics for GANs on the synthesis
task and demonstrate the improvement in the reconstructiontask. In particular, we found that simulating point-drops wasimportant for training GANs effectively.
For obstacle detection tasks, recent studies [9, 12] revealed
that simulating point-drops could mitigate the domain gap
between the real and simulated environments. Wu et al. [9]calculated the spatial prior from real LiDAR scans to sample
dropout noises. However, the sampled noise is independentof the instance. Manivasagam et al. [12] trained U-Net topredict point-drops as a binary classiﬁcation task. However,the drop prediction requires clean LiDAR data superimposedby multiple scans, and the classiﬁcation probability is not calibrated. In contrast, our approach only requires noisy LiDAR
data and can implicitly model the data-dependent uncertaintyof point measurability. As we discuss in Section V, ourlearned generator can be used to estimate realistic pointdrops from given observable points.
B. Modeling Image Noises
Some studies leveraged noisy training data for denoising
images [14] and synthesizing novel clean images [15]–[17].
Lehtinen et al. [14] eliminated various types of synthetic
noises, including multiplicative Bernoulli noise, which randomly masked a pixel to zero with a certain probability. Boraet al. [16] learned GANs from partially-zeroed images. We
assume that point-drops in LiDAR sensing can be categorized
as this type of noise. However, these studies [14, 16] assume
a predeﬁned noise model, such as pixel-level or patch-level
dropouts with a ﬁxed probability. Kaneko and Harada [15]proposed GAN architectures that can estimate noise distribution, but multiplicative binary noises are not covered. Li
et al. [17] learn the distribution of binary noises, however,the noise is independent of signals and approximated withsigmoid outputs.
As in [14, 16, 17], we assume multiplicative Bernoulli
noises to mimic the point-drops on the LiDAR scan map.
However, we do not set the probability; instead, we aim
to learn the pixel-wise probability model. Speciﬁcally, in
our case, the noise distribution type is deﬁned, but thenoise level and the generative relationship to the depth
modality are unknown owing to the complex physical factors
in measurement. Meanwhile, generating binary masks with
Bernoulli distribution is not differentiable, which means
inability to learn the parameterized probability model bybackpropagation. Therefore, this study employs the GumbelSoftmax trick [18, 19], a reparametrization approach fordiscrete sampling.
III. O UR A PPROACH
A. Data Representation
This paper assumes a bijective image representation [11],which is directly acquired from horizontal scanning ofmultiple laser receptors aligned vertically. For example, a
LiDAR that emits 𝑊 pulses for 𝐻 elevation angles produces
𝐻 × 𝑊 points with measured distances 𝑥, which can beconsidered as a cylindrical depth map with a size of 𝐻 × 𝑊.
An existing study [11] trained GANs with the depth map
representation. However, we found it difﬁcult to learn stably
without their preprocessing that narrows the spatial range ofthe depth map. Instead, we propose another approach for data
representation motivated by monocular depth estimation. In
a typical setting of LiDARs, most pixels of the acquired
depth map represent a near-to-mid range, and most of the
dynamic range is occupied by few distant points in a longtail distribution. Therefore, to gain the dynamic range of theMeasurability #!
Generator,"

Pixel-wise perturbation
Gumbel-Sigmoid!!
Inverse depth "Point-drop indicator $!

Real scan "#$%&
Discriminator~

Point-drop const %Synthetic scan "!

Realnessscalar

Masked blending

Latent code& ∼ ( 0, +!!
Inverse depth "

3'

$! ⨀!
"! + (1 − $! )⨀%Fig. 2. Overview of our proposed GAN framework. Generator 𝐺 𝜃 produces the inverse depth map 𝒙˜ 𝒛 and the corresponding measurability map 𝝅𝒛 ,from the sampled latent code 𝒛 ∼ N (0, 𝑰 ). The binary mask 𝒎𝒛 indicating point-drops is sampled from the measurability map 𝝅𝒛 . Discriminator 𝐷 𝜙distinguishes the processed inverse depth map 𝒙𝒛 from the real data 𝒙real .

majority regions, we transform the LiDAR depth map into an
inverse form 1/𝑥, i.e., an inverse depth map. We will discuss
how we can synthesize the cylindrical inverse depth maps
throughout this paper.

measurability map 𝝅𝒛 .
𝒎 𝒛 ∼ Bernoulli (𝝅𝒛 )(3)

The ﬁnal synthetic scan 𝒙 𝒛 is produced by masking thedense inverse depth 𝒙˜ 𝒛 using the binary mask 𝒎 𝒛 :B. Decomposed Synthesis of LiDAR Scans
A GAN typically consists of two networks: a generator
𝐺 𝜃 and a discriminator 𝐷 𝜙 , where 𝜃 and 𝜙 are trainable
parameters. In image synthesis tasks, 𝐺 𝜃 maps a latentvariable 𝒛 ∼ 𝑁 (0, 𝑰) to an image 𝒙 𝒛 = 𝐺 𝜃 (𝒛), whereas
𝐷 𝜙 distinguishes the generated image 𝒙 𝒛 from sampled real
images 𝒙real . The networks are trained in an alternatingfashion by minimizing the adversarial objective, e.g., thefollowing non-saturating loss [2]:L 𝐷 = −E 𝑥 [log 𝐷 𝜙 (𝒙real )] − E𝑧 [log(1 − 𝐷 𝜙 (𝐺 𝜃 (𝒛)))]
L𝐺 = −E𝑧 [log 𝐷 𝜙 (𝐺 𝜃 (𝒛))].

(1)
(2)

Here, 𝐺 𝜃 is modeled as a decoder convolutional networkthat gradually upscales the spatial resolution toward the ﬁnal
image. Although synthesizing natural images [3] has beensuccessfully achieved, we found it difﬁcult to learn noisy
depth maps with a stack of naı̈ve convolutions.
Let us consider modeling a scene where a vehicle ismoving, for example. If all the points are observable on
the LiDAR receptor, the shape of the vehicle would besmoothly morphed on the depth map. In this case, generating
a sequence of the depth maps is relatively easy if we move asampling point in the continuous data space by manipulating
a set of convolution kernels. However, in practice, somepoints are randomly dropped at the pixel level becauseof complex physical factors (such as mirror diffusion andmaterial reﬂectance). In these cases, the scattered point-drops
make the morphing discrete and cause learning difﬁculty.
Our key idea is to simulate the point-drop phenomenon by
a probabilistically invertible function to learn the underlying
manifold of smooth depth separately.
To this end, we introduce a decomposed depth map
representation into adversarial training. An overview of ourapproach is depicted in Fig. 2. First, we assume a dense
inverse depth map 𝒙˜ 𝒛 ∈ R 𝐻 ×𝑊 and a measurability map
𝝅𝒛 ∈ R 𝐻 ×𝑊 that represents the probability of laser reﬂection(e.g., mirror-like materials have low conﬁdence in returning
lasers due to the diffuse reﬂection, which results in pointdrops). We sample a binary mask 𝒎 𝒛 ∈ {0, 1} 𝐻 ×𝑊 from the𝒙 𝒛 = 𝒎 𝒛 ⊙ 𝒙˜ 𝒛 + (1 − 𝒎 𝒛 ) ⊙ 𝛼,

(4)where ⊙ is an element-wise product, and 𝛼 is a constantvalue representing the point-drop.
Note that sampling 𝒎 𝒛 is not differentiable, and thegradients cannot be propagated downstream. Therefore, we
reparameterize 𝒎 𝒛 with the straight-through (ST) GumbelSigmoid1 distribution [18, 19] to estimate the gradients. We
ﬁrst introduce the continuous relaxation 𝒎˜ 𝒛 as follows:
𝒆 + 𝒈 − 𝒈 𝝅𝒛12, 𝒆 = ln,(5)𝒎˜ 𝒛 = sigmoid𝜏1 − 𝝅𝒛where 𝒆 is a logit of 𝝅𝒛 and 𝒈1 , 𝒈2 ∈ R 𝐻 ×𝑊 are i.i.d.
samples from Gumbel(0, 𝑰), which perturbs 𝒆 at the pixel
level. Moreover, 𝜏 is a hyperparameter called temperature,
which controls the slope of the sigmoid function. With alow value of 𝜏, the soft mask 𝒎˜ 𝒛 approaches a binary mask,
but the variance of the gradients is large. Finally, the soft
mask 𝒎˜ 𝒛 is discretized into the binary mask 𝒎 𝒛 at eachpixel location (𝑖, 𝑗) as follows:(𝑖, 𝑗
1 𝒎˜ 𝒛 ≥ 0.5
𝑖, 𝑗
𝒎𝒛 =
(6)
𝑖, 𝑗
0 𝒎˜ 𝒛 < 0.5

ST Gumbel-Sigmoid [18, 19] approximates this thresholding by an identity function to enable gradient propagation:i.e., we use the stochastic binary mask 𝒎 𝒛 in the forward
step, and we approximate the gradients by 𝒎˜ 𝒛 during the
backward step. In summary, instead of modeling raw data
directly, our generator 𝐺 𝜃 aims to jointly produce the dense
inverse depth 𝒙˜ 𝒛 and the associated conﬁdence of point
measurability 𝝅𝒛 . Discriminator 𝐷 𝜙 only determines therealness of the processed inverse depth map 𝒙 𝒛 in Eq. 4.
C. Multilevel Gumbel Sampling
As introduced above, the pixel-level Gumbel sampling
offers the differentiable binarizer. The stochastic behavior
can learn the pixel-level uncertainty. However, we observed
another level of uncertainty appear in real LiDAR data. For
1 Gumbel-Sigmoid is a binary case of Gumbel-Softmax.

TABLE I

Pixel-synchronized

BASELINE ARCHITECTURE . 𝐻 AND 𝑊 ARE THE HEIGHT AND WIDTH OF
THE INVERSE DEPTH MAP, RESPECTIVELY. †T HE SHAPE IS REPLACEDWITH 2 × 𝐻 × 𝑊 FOR DUS TY-I AND 3 × 𝐻 × 𝑊 FOR DUS TY-II.

Point-drop indicator~)+,
Measurability #!

~Masking

Nonlinearity

Output shape

Generator 𝐺 𝜃

Generator output

Fig. 3.

Pixel-independent(Input)–Transposed Conv. 𝐻 /16 × 𝑊 /16
Transposed Conv.
4 × 4
Transposed Conv.
4 × 4
Transposed Conv.
4 × 4
Transposed Conv.
4 × 4–
Leaky ReLU
Leaky ReLU
Leaky ReLU
Leaky ReLU
Tanh

512 × 1 × 1
512 × 𝐻 /16 × 𝑊 /16
256 × 𝐻 /8 × 𝑊 /8
128 × 𝐻 /4 × 𝑊 /4
64 × 𝐻 /2 × 𝑊 /2
1 × 𝐻 × 𝑊†

Discriminator 𝐷 𝜙()*

Measurability #!

Layer(Input)–Blur ﬁltering [15]3 × 3Convolution
4 × 4
Convolution
4 × 4
Convolution
4 × 4
Convolution
4 × 4
Convolution
𝐻 /16 × 𝑊 /16––
Leaky ReLU
Leaky ReLU
Leaky ReLU
Leaky ReLU–

1 × 𝐻 × 𝑊
2 × 𝐻 × 𝑊
64 × 𝐻 /2 × 𝑊 /2
128 × 𝐻 /4 × 𝑊 /4
256 × 𝐻 /8 × 𝑊 /8
512 × 𝐻 /16 × 𝑊 /16
1 × 1 × 1

Inverse depth "!!

Multilevel Gumbel sampling. ⊙ denotes an element-wise product.

example, on the depth map of KITTI, some cars have pointdrops on only the transparent windows, while others are
missing most of the body. We assume colors and materials
cause this object-level multimodality of point drops even forthe same shape. The pixel-level Gumbel sampling is hard tocatch this distribution, and as a result, the variation would be
leaked into the latent space. We introduce a simple extension
to separate the object-level uncertainty from the latent space.
Revisiting Eq. 5, we made pixel-independent perturbations
to model the measurability. Here, we call the measurability
pix
pix
and the sampled mask as 𝝅𝒛 and 𝒎 𝒛 , respectively. We
img
now consider another measurability branch 𝝅𝒛 to be added
pixel-synchronized perturbations, where we sample scalars
𝑔1 , 𝑔2 ∼ Gumbel(0, 1) in training and cancel them in testingimg
to switch to a deterministic mask 𝒎 𝒛 . The ﬁnal mask 𝒎 𝒛
img
pix
is determined by element-wise product 𝒎 𝒛 ⊙ 𝒎 𝒛 .
IV. S YNTHESIS BY L ATENT S AMPLING
We ﬁrst report the synthesis performance of our proposed
methods by evaluating the similarity between the distributions of real data and sampled synthetic data. We further
discuss the reconstruction performance in Section V.
A. Dataset
We built the inverse depth maps from two types of LiDAR
datasets: KITTI [1] and MPO [13], which have different
vertical resolutions and scenes. The LiDARs of both the
KITTI and MPO datasets have the same distance range of(0.9𝑚, 120𝑚), where we take the reciprocal of the distancesand normalize the range into (−1, 1) for the training stabilityof GANs. When reconstructing point clouds from the inverse
depth map, we use the per-pixel average angles calculated
from the training set.
1) KITTI: We used the KITTI odometry [1] dataset, which
is a benchmark for odometry evaluation. The dataset provides
22 trajectories of LiDAR scans measured by the Velodyne
HDL-64E2 scanner, where each scan has 64 layers vertically.
It includes 19,310 scans for training, 4,071 scans for validation, and 20,351 scans for testing. The provided data areprocessed sequences of Cartesian coordinate points orderedby azimuth and elevation angles. Most studies applied spherical projection to create a depth map from the point sets.
However, this approach produces sampling artifacts due tothe scanner’s nonlinear vertical spacing3 . As in Caccia et

Kernel size

al. [11], we ﬁrst chunk the ordered sequence into 64 subsequences, where each represents one elevation angle. We
then subsample 256 points for each sub-sequence and stackthem to form a 64 × 256 inverse depth map. Unlike [11], we
do not clip the vertical angle and the distance range and do
not ﬁll in the point-drops with adjacent pixels4 .
2) MPO: We also use the Multimodal Panoramic 3D
Outdoor dataset (MPO) [13]. In particular, we use a “sparse”
dataset of MPO, which includes a total of 34,200 LiDAR
scans from 60 trajectories. The dataset has diversity in geometry and point-drop distribution because it was constructed
to classify six different outdoor scenes: coast, forest, indoor
parking, outdoor parking, residential area, and urban area.
Each scan is a sequence of Cartesian coordinate points
with vertical angle IDs, obtained using Velodyne HDL-32E5
scanner. In this study, we split the trajectories into 20,322
scans for training, 3,787 scans for validation, and 10,091
scans for testing. We chunk the sequence according to thevertical angle ID and create a 32 × 256 inverse depth map
similar to the KITTI procedure.
B. Models
1) Baseline: Table I shows the network architectures ofour baseline GAN. The architecture design is inspired by theexisting work [11]. However, we modify the output shapefor our task, remove all normalization layers, and only use
the Leaky ReLU nonlinearity with a negative slope of 0.2.
We add vertical/horizontal blur ﬁltering [15] as the ﬁrst
layer of the discriminator, to mitigate the learning difﬁcultyof a discrete distribution. As in existing studies [20, 21],
we replace the zero-padding of all convolutional layerswith horizontal circular padding, where the right and left
boundaries are padded with the pixels on the opposite sides.
It enables the model to spread the receptive ﬁelds out of theboundaries so that it can process the cylindrical tensors.
2) DUSty-I (ours): For a fair comparison, in this method,
we only modify the last convolutional layer of the baseline

2 https://velodynelidar.com/products/hdl-64e/
3 The HDL-64E scanner has several versions with different verticalspacing, but it was not uniﬁed in studies with KITTI.

4 https://github.com/pclucas14/lidar_generation
5 https://velodynelidar.com/products/hdl-32e/

generator. The last layer produces 2-channel outputs forinverse depth and measurability. The tanh nonlinearity isapplied to only the inverse depth output. The measurability
output is transformed into a binary mask with pixel-level
Gumbel sampling, as introduced in Section III-B.
3) DUSty-II (ours): In this method, we add an extra
channel to the measurability output of the DUSty-I model todecompose the Gumbel sampling into pixel-level and imagelevel, as introduced in Section III-C. The same temperature𝜏 is used for both pixel-level and image-level Gumbel
sampling.
C. Implementation Details
We set the maximum distance to the point-drop constant
𝛼 in Eq. 4. Moreover, we observed that a low value oftemperature 𝜏 in Eq. 5 slowed the convergence despite abetter approximation of the binary masks. This study uses aﬁxed 𝜏 = 1 in all experiments. For Gumbel sampling frommeasurability 𝝅 in Eq. 5, our network directly outputs thelogit 𝒆, instead of 𝝅 = sigmoid(𝒆). For adversarial training,
we use the non-saturating loss in Eq. 1 and 2, with a 𝑅1
gradient penalty [3]. The penalty coefﬁcient was set to 1.
All parameters were updated by Adam [22] optimizer for
25M iterations with a learning rate of 0.002 and a batch
size of 32. We apply the equalized learning rate [4] for all
trainable layers, where the parameters are initialized withN (0, 1) and scaled by He’s initialization constant at runtime.
Moreover, we apply DiffAugment by Zhao et al. [23], which
composed of color, translation, and cutout augmentations to
the discriminator inputs. For the translation augmentation,
we circulate the inputs horizontally. DiffAugment was not
required for training stability but greatly improved the qualityof inverse depth maps. We take the exponential moving
average for generator parameters 𝜃. We implemented our networks in PyTorch and performed distributed training on two
NVIDIA Titan RTX GPUs. The training required approximately 22 hours for each model. The code will be available athttps://github.com/kazuto1011/dusty-gan.
D. Evaluation Metrics
We measured four types of distributional similarities between the sets of reference and generated point clouds [5]:Jensen–Shannon divergence (JSD) for quality, coverage(COV) for diversity, minimum matching distance (MMD)
for quality, and 1-nearest neighbor accuracy (1-NNA) for
both quality and diversity evaluation. We found that thesynthesis evaluation on LiDAR point clouds had an extremely high cost, particularly for calculating the distancematrix for the sets of point clouds in COV, MMD, and
1-NNA. For efﬁciency, we ﬁrst subsampled the series oftest data to be 5,000 data in total, and for each, we also
randomly subsampled 2,048 points from the full 𝐻 × 𝑊
points by farthest point sampling. We generated 5,000 datafrom each model and reduced the number of points in thesame manner. We used the Chamfer distance to measure thepairwise similarity for the distance matrix of point clouds.
Additionally, we computed the sliced Wasserstein distance(a) Scores on KITTI validation set(b) Scores on MPO validation set
Fig. 4. Impact of relative tolerance 𝛽 for the baseline method. We optimized
𝛽 based on the weighted score and set 0.008 for KITTI and 0.0065 for MPO(dotted vertical lines).

(SWD) [4] to measure the patch-based image similarity
for the inverse depth maps. For all metrics, we report themean scores with the standard deviation over ﬁve runs withdifferent latent codes.
E. Quantitative Results
It is non-trivial for the baseline model to produce values
that exactly match the point-drop constant 𝛼. Therefore, we
ﬁrst optimize a relative tolerance 𝛽 to decide if an interest𝑖, 𝑗pixel 𝒙 𝒛 is 𝛼 based on𝑖, 𝑗k𝒙 𝒛 − 𝛼k≤ 𝛽,𝑥max − 𝑥min

(7)

where 𝑥max − 𝑥min is a range of the inverse depth. 𝛽 is
optimized by HyperOpt [24] within [10−3 , 10−1 ] for 100
steps to minimize the following weighted 3D score, JSD ×
10 − COV + MMD × 102 + 1-NNA, on the validation set foreach dataset. For efﬁciency, we reduce the number of points
to 512. Fig. 4 shows trade-off results because the point-drop
tolerance erodes the available range of inverse depth. We
employ 𝛽 = 0.008 for KITTI and 𝛽 = 0.0065 for MPO. We
note that our noise-aware model DUSty does not require thetolerance, i.e., 𝛽 = 0.
In Table II, we compare the synthesis performance among
the baseline and our methods. On both KITTI [1] andMPO [13] datasets, our methods outperformed the baselineon all the 3D metrics. On the SWD score, the baseline wasbetter than DUSty-I on KITTI, while our methods outperformed in the other cases. We consider that our samplingbased binary mask still has an “appearance” gap with thetruth distribution of point-drops compared with the directmodeling by the baseline. Meanwhile, the 3D results onfour metrics indicate that our decomposition approach has apositive effect on the disentangled modeling of inverse depth
and point-drop.
F. Qualitative Results
Fig. 5 compares inverse depth maps generated from thebaseline and our DUSty-I model on the KITTI dataset.
Our method succeeds in representing the sharp jump edgesaround point-drops. However, the baseline has interpolated
pixels that can be unexpected noise in point clouds. Moreover, we show some synthetic examples of all methods forTABLE II
Q UANTITATIVE COMPARISON OF SYNTHESIS PERFORMANCE . ↓: THE LOWER THE BETTER . ↑: THE HIGHER THE BETTER .
Dataset

KITTI

MethodJSD ×102 ↓(3D quality)

COV [%] ↑(3D diversity)

MMD ×103 ↓(3D quality)

1-NNA [%] ↓(3D quality/diversity)

SWD ↓(2D quality)

Baseline
DUSty-I (ours)
DUSty-II (ours)

6.45 ± 0.06
2.85 ± 0.01
3.54 ± 0.07

4.99 ± 0.04
38.08 ± 0.55
38.05 ± 0.65

2.36 ± 0.03
1.14 ± 0.01
1.12 ± 0.01

99.99 ± 0.00
93.69 ± 0.19
94.62 ± 0.30

0.158 ± 0.011
0.167 ± 0.010
0.151 ± 0.011

0.93

35.02

0.87

96.72

0.182

2.53 ± 0.03
1.47 ± 0.02
1.71 ± 0.02

6.10 ± 0.12
22.85 ± 0.34
30.94 ± 0.34

2.03 ± 0.01
1.53 ± 0.01
1.54 ± 0.01

99.31 ± 0.03
95.03 ± 0.16
94.84 ± 0.19

0.180 ± 0.013
0.174 ± 0.014
0.160 ± 0.018

0.74

34.88

1.52

87.09

0.158

Training set

MPO

Baseline
DUSty-I (ours)
DUSty-II (ours)
Training set

xz

xreal
(a) Training data

Zoom

xz
(b) Baselinex̃ z

Point cloudsπzxz(c) DUSty-I (ours)x̃ z(a) Baseline(b) DUSty-I (ours)

Fig. 5.
Qualitative comparison of inverse depth outputs 𝒙𝒛 , zoomedin regions (white boxes in 𝒙𝒛 ), and points clouds, with and without the
proposed measurability learning. Best viewed in color.

pix

πz

img

πz

xz
(d) DUSty-II (ours)

KITTI in Fig. 6 and MPO in Fig. 7. We can see that
our methods DUSty-I and DUSty-II successfully learned acomplete inverse depth maps in 𝒙˜ 𝒛 and drop uncertainty
in 𝝅𝒛 . Regarding 𝝅𝒛 , the results show that the sampling
variation would occur on the boundaries of vehicle-likeobjects and the scattered noises. In contrast, the steady pointdrops are represented as sufﬁciently low measurability, suchas the ego-vehicle shadows in KITTI and the sky regionsin MPO. Furthermore, DUSty-II separately learned the pixel
img
correlation of the drop uncertainty in 𝝅𝒛 .
V. R ECONSTRUCTION BY L ATENT S PACE E XPLORATIONWith trained GANs, we can reproduce the given data by
optimizing a latent code [3, 25], where the task is called GAN
inversion. In this section, we optimize the latent code 𝒛 toreconstruct unseen LiDAR data from KITTI and MPO test
sets. We demonstrate our decomposition approach is easier
to ﬁnd a matching latent code.
A. Objective
To reproduce a given inverse depth 𝒙target , we optimize thelatent code 𝒛 by minimizing the L1 distance between 𝒙 target

Fig. 6. Qualitative comparison on the KITTI [1] dataset. Each column
is from a unique latent code. 𝒙real and 𝒙𝒛 denotes real LiDAR data andgenerated LiDAR data, respectively. 𝒙˜ 𝒛 and 𝝅𝒛 denotes the inverse depth
and measurability outputs from our proposed models, respectively.

and generated inverse depth 𝒙˜ 𝒛 before masking:
Í
𝑖, 𝑗
𝑖, 𝑗
𝑖, 𝑗
𝑖, 𝑗 1[𝒙 target ≠ 𝛼] k𝒙 target − 𝒙˜ 𝒛 k 1Í,𝒛ˆ = arg min𝑖, 𝑗𝒛𝑖, 𝑗 1[𝒙 target ≠ 𝛼](8)where 1[𝒙target ≠ 𝛼] ∈ {0, 1} 𝐻 ×𝑊 is a point-drop indicatorof the target data 𝒙 target , and (𝑖, 𝑗) is the 2D location of theinverse depth map. The latent 𝒛 ∈ R𝑑 is updated for 1,000
iterations by the Adam optimizer with a learning rate of 0.1.
Considering the nature of high-dimensional Gaussian priors,
the search space of the latent 𝒛 √is constrained to a surface ofa hypersphere with a radius of 𝑑 [25]. To avoid sticking in
local minima, we add a Gaussian noise 𝜖 ∼ N (0, 0.05𝑡 2 𝑰),
where 𝑡 goes from 1 to 0 in iterations [3]. The optimization
required approximately 4 seconds for each sample.
B. Quantitative Results
To assess the reconstruction performance, we compute
standard metrics in depth estimation [1]: relative absolutexreal(a) Training data

hand, we can also observe that the far objects and thin
structures are still difﬁcult to reconstruct in some cases.

xz(b) Baseline
x̃ z
πz
xz
(c) DUSty-I (ours)
x̃ z
pix

πz

img
πz

xz
(d) DUSty-II (ours)

Fig. 7. Qualitative comparison on the MPO [13] dataset. Each column isfrom a unique latent code.
TargetBaseline
DUSty-I
DUSty-II

Fig. 8. Qualitative comparison of the reconstructed inverse depth maps.
Our method is better at reproducing the geometric features of the given
targets and generating realistic point-drops.

VI. C ONCLUSION
In this paper, we proposed DUSty, a GAN frameworkfor LiDAR scan synthesis based on the decomposed image
representation of inverse depth and point-drop. Our method
showed effectiveness on both synthesis and reconstruction
tasks compared to the baseline that directly models the
LiDAR scans. The reconstruction experiments indicated potential applications of our method, such as restoration ofcorrupted data and upsampling of sparse scans from lowcost LiDARs. Our method would also be applicable to rendering realistic point-drops for fully measurable simulation
data [9, 12] since our pixel-wise depth reconstruction cansimultaneously produce a measurability map. Although thisstudy used standard GAN architectures, further investigationon the network design could improve the generation quality.
Future work includes introducing our point-drop learning
into the other large networks [3, 4] and applying our synthetic data to perception tasks.
ACKNOWLEDGMENT
This work was supported by a Grant-in-Aid for JSPS
Fellows Grant Number JP19J12159 and JSPS KAKENHI
Grant Number JP20H00230.
R EFERENCES

error (Abs Rel), relative squared error (Sq Rel), root meansquared error (RMSE), RMSE on logarithmic depth (RMSElog), and the ratio of pixels with relative error 𝛿 under thethreshold (𝛿 < 1.25𝑖 , 𝑖 = 1, 2, 3). Moreover, we also compute
the Chamfer distance (CD) between point clouds from 𝒙 target
and 𝒙 𝒛ˆ . Table III shows the results of the baseline and our
proposed methods. Our method outperformed the baseline on
all the metrics, and DUSty-I was slightly better than DUStyII in most cases.
C. Qualitative Results
Fig. 8 compares the reconstructed inverse depth maps 𝒙 𝒛ˆfrom the baseline and our methods. The baseline failed toreconstruct some foreground objects ( 𝒛ˆ ← 𝒙target ), contrary
to the synthesis results in Fig. 5 and 6 (𝒛 → 𝒙 𝒛 ). In contrast,
our method reproduced the details and also synthesized more
realistic point-drops. This indicates that our decomposition
trick yields a better embedding of the geometric styles of
LiDAR scans.
D. Applications
In Fig. 9, we show some examples from DUSty-I, where
the targets are with three types of strong corruptions: 90% of
points are randomly dropped (Fig. 9(b)), 8 out of 64 horizontal lines are observable (Fig. 9(c)), and pixel-wise noise fromN (0, 0.01) are added to the depth values (Fig. 9(d)). Despite
very sparse or noisy targets 𝒙target , our method succeeded inreconstructing inverse depth maps with underlying complete
surfaces 𝒙˜ 𝒛ˆ and rendering realistic point-drops. On the other[1] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics:
The kitti dataset,” The International Journal of Robotics Research(IJRR), vol. 32, no. 11, pp. 1231–1237, 2013.
[2] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,
S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,”
in Advances in Neural Information Processing Systems (NIPS), 2014,
pp. 2672–2680.
[3] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila,“Analyzing and improving the image quality of stylegan,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), 2020, pp. 8110–8119.
[4] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of
GANs for improved quality, stability, and variation,” in Proceedingsof the International Conference on Learning Representations (ICLR),
2018.
[5] G. Yang, X. Huang, Z. Hao, M.-Y. Liu, S. Belongie, and B. Hariharan,“Pointﬂow: 3d point cloud generation with continuous normalizing
ﬂows,” in Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), 2019, pp. 4541–4550.
[6] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas, “Learning representations and generative models for 3D point clouds,” inProceedings of the International Conference on Machine Learning(ICML), 2018, pp. 40–49.
[7] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li,
S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu,“ShapeNet: An information-rich 3D model repository,” Tech. Rep.,
Dec. 2015.
[8] B. Li, T. Zhang, and T. Xia, “Vehicle detection from 3d lidar using
fully convolutional network,” arXiv preprint arXiv:1608.07916, 2016.
[9] B. Wu, A. Wan, X. Yue, and K. Keutzer, “SqueezeSeg: Convolutional
neural nets with recurrent crf for real-time road-object segmentationfrom 3d lidar point cloud,” in Proceedings of the IEEE International
Conference on Robotics and Automation (ICRA), 2018, pp. 1887–
1893.
[10] Y. Zhang, Z. Zhou, P. David, X. Yue, Z. Xi, B. Gong, and H. Foroosh,“Polarnet: An improved grid representation for online lidar point
clouds semantic segmentation,” in Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR),
2020, pp. 9601–9610.

TABLE III
Q UANTITATIVE COMPARISON OF RECONSTRUCTION PERFORMANCE . ↓: THE LOWER THE BETTER . ↑: THE HIGHER THE BETTER .
3D error ↓2D error ↓Abs RelSq Rel

RMSE

RMSE log

𝛿 < 1.25

𝛿 < 1.252

𝛿 < 1.253

DatasetMethodKITTI

Baseline
DUSty-I (ours)
DUSty-II (ours)

5.31
1.64
1.66

0.199
0.074
0.077

12.394
0.555
0.546

9.667
3.722
3.663

0.280
0.155
0.157

89.11
92.09
91.91

95.21
96.94
96.93

97.51
98.69
98.69

MPO

Baseline
DUSty-I (ours)
DUSty-II (ours)

9.33
1.25
1.28

0.197
0.109
0.111

9.008
0.878
1.013

10.523
4.930
5.079

0.308
0.229
0.232

84.15
85.90
85.83

92.26
94.02
93.89

95.61
97.11
97.00

(b) 90% dropout (c) 1/8 scan lines(d) Gaussian

Point clouds xtarget

TargetPoint clouds xtarget
Reconstruction
Point clouds x ẑ x̃ ẑ(a) Original

Reconstruction
Point clouds x ẑ x̃ ẑ(a) Original

Target2D accuracy [%] ↑CD ×104(b) 90% dropout (c) 1/8 scan lines(d) GaussianFig. 9. Reconstruction examples for the target LiDAR scans 𝒙target with three types of corruption. All results were generated by our DUSty-I trainedon KITTI. (a) Sample without any corruption from the KITTI test set. (b) 90% points are randomly dropped. (c) Only 8 out of 64 horizontal lines areobservable. (d) An additive Gaussian noise from N (0, 0.01) is applied to a depth value.

[11] L. Caccia, H. van Hoof, A. Courville, and J. Pineau, “Deep generativemodeling of lidar data,” in Proceedings of the IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), 2019, pp. 5034–
5040.
[12] S. Manivasagam, S. Wang, K. Wong, W. Zeng, M. Sazanovich,
S. Tan, B. Yang, W.-C. Ma, and R. Urtasun, “Lidarsim: Realistic
lidar simulation by leveraging the real world,” in Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 2020, pp. 11 167–11 176.
[13] O. Martinez Mozos, K. Nakashima, H. Jung, Y. Iwashita, and R. Kurazume, “Fukuoka datasets for place categorization,” The International
Journal of Robotics Research (IJRR), vol. 38, no. 5, pp. 507–517,
2019.
[14] J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala, and T. Aila, “Noise2Noise: Learning image restoration withoutclean data,” ser. Proceedings of Machine Learning Research, vol. 80.
Stockholmsmässan, Stockholm Sweden: PMLR, 10–15 Jul 2018, pp.
2965–2974.
[15] T. Kaneko and T. Harada, “Noise robust generative adversarial networks,” in Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), 2020, pp. 8404–8414.
[16] A. Bora, E. Price, and A. G. Dimakis, “AmbientGAN: Generativemodels from lossy measurements,” in Proceedings of the International
Conference on Learning Representations (ICLR), 2018.
[17] S. C.-X. Li, B. Jiang, and B. Marlin, “Misgan: Learning fromincomplete data with generative adversarial networks,” in Proceedingsof the International Conference on Learning Representations (ICLR),
2019.
[18] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization withgumbel-softmax,” in Proceedings of the International Conference on
Learning Representations (ICLR), 2017.
[19] C. J. Maddison, A. Mnih, and Y. W. Teh, “The concrete distribution:
A continuous relaxation of discrete random variables,” in Proceedingsof the International Conference on Learning Representations (ICLR),

2017.
[20] S. Schubert, P. Neubert, J. Pöschmann, and P. Protzel, “Circularconvolutional neural networks for panoramic images and laser data,”
in Proceedings of the IEEE Intelligent Vehicles Symposium (IV), 2019,
pp. 653–660.
[21] K. Nakashima, H. Jung, Y. Oto, Y. Iwashita, R. Kurazume, and O. M.
Mozos, “Learning geometric and photometric features from panoramic
lidar scans for outdoor place categorization,” Advanced Robotics,
vol. 32, no. 14, pp. 750–765, 2018.
[22] D. P. Kingma and J. Ba, “Adam: a method for stochastic optimization,”in Proceedings of the International Conference on Learning Representations (ICLR), 2015.
[23] S. Zhao, Z. Liu, J. Lin, J.-Y. Zhu, and S. Han, “Differentiable
augmentation for data-efﬁcient gan training,” in Advances in Neural
Information Processing Systems (NeurIPS), 2020.
[24] J. Bergstra, D. Yamins, and D. Cox, “Making a science of model
search: Hyperparameter optimization in hundreds of dimensions forvision architectures,” in Proceedings of the International Conference
on Machine Learning (ICML), 2013, pp. 115–123.
[25] S. Menon, A. Damian, S. Hu, N. Ravi, and C. Rudin, “PULSE: Selfsupervised photo upsampling via latent space exploration of generativemodels,” in Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), 2020, pp. 2437–2445.