TPU-GAN: Learning temporal coherence from dynamic point cloud sequences
Authors: Zijie Li
Tianqin Li
Amir Barati Farimani
Venue: ICLR 2022 Poster (2022)
Keywords: Point cloud super resolution
Temporal learning
Generative Adversarial Networks

Abstract: Point cloud sequence is an important data representation that provides flexible shape and motion information. Prior work demonstrates that incorporating scene flow information into loss can make model learn temporally coherent feature spaces. However, it is prohibitively expensive to acquire point correspondence information across frames in real-world environments. In this work, we propose a super-resolution generative adversarial network (GAN) for upsampling dynamic point cloud sequences, which does not require point correspondence annotation.  Our model, Temporal Point cloud Upsampling GAN (TPU-GAN), can implicitly learn the underlying temporal coherence from point cloud sequence, which in turn guides the generator to produce temporally coherent output. In addition, we propose a learnable masking module to adapt upsampling ratio according to the point distribution. We conduct extensive experiments on point cloud sequences from two different domains: particles in the fluid dynamical system and human action scanned data. The quantitative and qualitative evaluation demonstrates the effectiveness of our method on upsampling tasks as well as learning temporal coherence from irregular point cloud sequences.

Published as a conference paper at ICLR 2022
TPU-GAN: LEARNING TEMPORAL COHERENCE FROM DYNAMIC POINT CLOUD SEQUENCES
Zijie Li
Tianqin Li
Department of Mechanical Engineering
School of Computer Science Carnegie Mellon University
Cameggi Mellon University
Amir Barati Farimani
Department of Mechanical Engincering Carnegie Mellon University
ABSTRACT
Point cloud sequence an important data representation that provides flexible shape and motion information. Prior work demonstrates that incorporating scene flow information into loss can make model learn temporally coherent feature spaces. However, it is prohibitively expensive acquire point correspondence information across frames real-world environments. In this work, we propose a super-resolution generative adversarial network (GAN) for dynamic point cloud quences without requiring point correspondence annotation. Our model, Temporal Point cloud Upsampling GAN (TPU-GAN), can implicitly learn the underlying temporal coherence from point cloud sequence, which in turn guides the gener- ator to produce temporally coherent output. In addition, we propose a learnable masking module to adapt upsampling ratio according to the point distribution. We conduct extensive experiments on point cloud sequences from two different domains: particles in the fluid dynamical system and human action scanned data. The quantitative and qualitative evaluation demonstrates the effectiveness of our method on upsampling task well as learning temporal coherence from irregular point cloud sequences.
INTRODUCTION
Temporally evolving point clouds are ubiquitous in various fields, e.g. particles in the N-body system, real-world moving objects captured by LiDAR sensor or RGB-D camera. Understanding the spatiotemporal pattern lies in these sequential data are of great importance to engineering and scientific disciplines. However, high-resolution point cloud sequences can be very expensive obtain. In practice, high-resolution physics simulation demands substantial computational resources and 3D scanned data are often sparse and noisy due hardware constraints. Impressive progress has been made by applying learning-be based algorithms (Yu et al., 201 2018b; Li et al., 2019; Wu et al., 2019; Yifan al., 2019; Qian al., 2021) reconstruct dense and clean point cloud given the observed sparse ones. While these works provide powerful framework for upsampling static point clouds, the change of points with respect time is not taken into consideration. Prantl et al. (2020) demonstrates that supervising the scene flow (displacement of points) of generated points during the training process can greatly stabilize the learned latent space for point cloud super-resolution task. Yet this requires knowing the point correspondence across the frames to derive the scene flow which is expensive and sometimes impossible to acquire in real-world scenarios. For 3D scanned data, the point clouds in two frames do not necessarily have correspondences and the number of points may change.
In this case, generative adversarial networks (GANs) (Goodfellow et al., 2014) provides an attractive alternative for learning useful generative and discriminative representation without label and heuristic loss function (e.g. L2 loss that penalizes the scene flow difference). Under the context of point cloud sequence's 's super-resolution patterns from data implicitly. With these learned spatiotemporal discriminative representation, we can further use them guide the generator produce output that spatially and temporally coherent.
the discriminator to learn both the spatial and temporal

---

Published as a conference paper at ICLR 2022
More formally, in this work present GAN framework for super-resolution on the dynamic point cloud sequences. By experimenting on point cloud sequences from two very different domains fluid particles and 3D scanned human action, show that the proposed upsampling model can produce output that are spatially and temporally consistent with target data without explicit supervision on the point displacements across frames. We also show that the discriminative representation learned by temporal discriminator can be used for classification task and provides competitive performance. Furthermore, we propose learnable masking module adaptively upsample points that have irregular density distribution. This allows model to produce non-uniform point distribution that matches the target domain.
2 RELATED WORKS
Deep learning on static point clouds The pioneering work PointNet (Qi et al., 2017a) carries the success of deep neural networks on 2D structured grid data the field of unstructured 3D point cloud processing. PointNet proposes a neural network architecture with multi- perceptron (MLP) shared acrossed points and aggregation mechanism that respects permutation invariance in the input. Following PointNet, a wide array of works had contributed to improve and extend the neural representation learning paradigm on point cloud data (Qi et al., 2017b; Li et al., 2018; Wang et al., 2019b; Wu et al., 2020; Zhao et al., 2019; Shen et al., 2018). 8). These methods have shown to learn good representation for static point cloud analysis task such as classification, semantic segmentation and object detection.
Learning-based point cloud upsampling With the success in learning-based image super- resolution (Dong et al., 2015; Ledig et al., 2017), a number of works transfer the learning-based super-resolution concept to the 3D point cloud. PU-Net (Yu et al., 2018b) proposes the first learning- based method that upsamples point cloud features with multi-level expansion units. Yu et al. (2018a) extends and improves PU-Net by adding an edge-aware loss to consolidate the upsampled point clouds. Yifan et al. (2019) proposes a progressive point cloud upsampling network which supervises the upsampled point clouds at different levels. Li et al. (2019); Wu al. (2019) leverage GANs to learn high-quality upsampling result, which demonstrate adversarial training can benefit point cloud upsampling task. Very recent work Qian et al. (2021) proposes Inception DenseGCN and NodeShuf- fle, which significantly improves the upsampling quality. Another concurrent work PUGeo-Net (Qian et al., 2020) proposes a geometric-centric deep learning framework for learning the local fundamental forms.
Spatiotemporal representation learning on point clouds Different deep learning methodologies for learning discriminative spatiotemporal representation for video classification have been proposed (Ng et al., 2015; Donahue et al., 2016; Feichtenhofer et al., 2016; Carreira & Zisserman, 2018; Tran et al., 2015), and been further explored on the generative tasks including video generation (Tulyakov et al., 2017: Clark et al., 2019; Saito et al., 2020) and super-resolution (Xie et al., 2018; Wang et al., 2019a; Haris et al., 2019; Chu et al., 2020). Following previous approaches that treat video data as volumetric data, Choy et al. (2019); Luo et al. (2020) transform point cloud sequences into volumetric voxel-based representation and apply convolution neural networks on it. Liu et al. (2019b; 2020) extends the set convolution from PointNet++ (Qi et al., 2017b) by establishing neighborhood across different frames. PSTNet (Fan et al., 2021b) proposes a spatiotemporal decoupled point convolution for point cloud sequence processing. Recent works Fan et al. a)) Wang et al. (2021a) introduce attention mechanism to better capture data correlation across frames. In terms of generative tasks, Tranquil Clouds (Prantl et al., 2020) proposes a temporal loss that supervises the scene flow across frames and stabilize the generation of point clouds. Rempe et al. (2020) proposes CaSPR, a deep learning framework for learning continuous object-centric spatiotemporal representation from point clouds. Lu et al. (2021) proposes interpolation network to enhance the temporal resolution of point cloud sequence. Another recent work Wang al. (2021b) proposes a learning-bas based recurrent temporal alignment module for assimilating temporal information during upsampling sequential point clouds.
In general, our work is closely related to PointNet++ (Qi et al., 201 and MeteorNet (Liu et al., 2020) which use set abstraction with different grouping strategies to learn discriminative representation on point clouds. In the generator, we adopt a similar multi-scale feature extractor architecture as the one proposed in PU-GCN (Qian et al., 2021). Compared to PU-GAN (Li et al., 2019), our method takes the change of points in time dimension into account, and uses hierarchical structures in the discriminator instead of operating on full resolution point clouds. Under the hood of sequential point cloud data generative modelling, Tranquil Clouds (Prantl et al., 2020), CaSPR (Rempe et al.,
2

---

Published as a conference paper at ICLR 2022
(a) Generator
Nx
X'x.3
N'
peampling Medule
7
Festure Extractor
Residuals
Rearaege
Renovz daplicate points
NX!
Input
3
Encoding
Masking Medule
Output
Masked
Duplicate
Mask
(b) Temporal Discriminator
(c) Spatial Discriminator
3
XX/ & J.C,
[-]
Set Alistraction
r
w//k XC3
1x
D Inpot
IXC
Set. Abstraction
Set Abstraction
Flow Module
0
Set Abstraction
0
L. layers
Sei Alsttrction
Ixl Confidence
1x1
MIP
MEP
Input
X layers
Confidence
Figure 1: Overview of TPU-GAN's architecture, it comprises generator, temporal discriminator and a spatial discriminator. (a) The Generator takes coarse point cloud as input and predicts refined point cloud. Inside the generator, the input is first project into high-dimensional latent encoding, the encoding is then transformed into residuals by Upsampling Module and binary mask by Masking Module. Through an element-wiss wise multiplication O residuals of unnecessary upsampling will be masked to zeros. (b) The Temporal Discriminator takes three consecutive frames point clouds as input, extracts features via set abstraction layesr and flow embedding layers, and predicts the confidence if temporally coherent. (c) The Spatial Discriminator takes a point cloud as input, extracts features via abstraction layers, and outputs the confidence if it is spatially coherent. (Detailed schematic of each sub-module can be found in Appendix A.
2020), and SPU (Wang et al., 2021b) share similar target with our work. CaSPR focuses on object- centric tasks and thus is not available for processing multi-object collections or scenes. Tranquil Clouds requires point correspondence across frames to evaluate the first-order and second-order finite difference loss, which our method does not require during the training phase. Compared to SPU (Wang et al., 2021b), we avoid processing the high resolution outputs multiple times by using temporal discriminator instead of recurrent architectures to learn the temporal coherence.
METHODS
3.1 HIERARCHICAL SPATIAL FEATURE LEARNING
Set abstraction Point clouds are irregular and unordered data where traditional convolution opera- tions usually do not fit. To cope with this irregularity, we adopt the set abstraction layer from Qi et al. (2017a;b) to extract features from point clouds for discriminative models in our proposed framework. Inside a set abstraction layer, point cloud divided into different groups, and then a universal set function approximator is applied extract the local features of each group
F
( max h(F5.r, -r,) JEN(()
where F denotes features of points, r denotes the positions of points, (ii)) denotes the neighborhood of point i, y and f are learnable functions which are usually implemented multi-layer perceptrons (MLPs).
Multi-scale graph convolution networks Recent work PU-GCN (Qian et al., 2021) proposes a based architecture (Inception DenseGCN and NodeShuffle) and demonstrates the superiority of using GCN on the point cloud upsampling tasks. Similar to the set abstraction layer from PointNet, the GCN layer in PU-GCN divides the point cloud into different groups and applies learnable functions
3

---

Published as a conference paper at ICLR 2022
these groups extract local features. The major differences are that first, the GCN layer in PU-GCN constructs the neighborhood following DGCNN's ( (Wang et al., 2019b) formulation by dynamically querying k-nearest neighbors in the latent space instead of Euclidean space, and second, in PU-GCN, each layer aggregates features from different scales of neighborhood at each layer. These differences make the Inception DenseGINN layer in PU-GCN more effective extracting features from different scales and improve the overall performance of model. However, since it is more computationally intensive than the standard set abstraction, we only apply similar GCN layers the generator, where it is most effective.
3.2 TEMPORAL FEATURE LEARNING
To capture the underlying patterns of point cloud sequences over the time dimension, we construct module that operates on the temporal neighborhood. We adapt the flow embedding layer from Liu al. (201 9b; 2020) to operate on neighborhood across frames. Given two consecutive frames of point clouds , xx++1 }
and their corresponding features {F", Fr++), }. we use the flow embedding layer to mix points across frames and extract temporal features. The mixture of features {!* pt+1] across two consecutive frames via flow embedding layer is defined as:
Fj
=
max h(F) Ff+1 .rj -1:) (EN(()
where j denotes the index of points from frame + 1 and denotes the index of points from frame t.
3.3 ENFORCING TEMPORAL COHERENCE VIA TEMPORAL DISCRIMINATOR
When the target data point cloud sequences that come from physical processes like particle dynamics or camera-scannel human action, these data are highly correlated in the time dimension. One way capture this temporal coherence would be directly using the scene flow information in the ground truth data (i.e. displacements of points) to supervise the training process, i.e. adding an additional loss term that evaluates the scene flow difference between ground truth and prediction. However, for many real-world scenarios, there are often no clear correspondences between point clouds from two frames. Therefore, the displacement of points between different frames are not annotated and cannot be used for training. To enforce the temporal coherence without manually encoding scene flow information, we propose a discriminator learn the temporal pattern from ground truth data implicitly and use the discriminator to guide generator predicting temporally coherent output.
Our proposed temporal discriminator is built upon set abstraction layer and flow embedding layer discussed in the previous section. The temporal discriminator takes three consecutive frames of point clouds as input. These point clouds are first downsampled and transformed by several layers of set abstraction. Then we apply the flow module, which consists of two flow embedding layers, to assimilate temporal information from the point cloud sequences. Finally, we use set abstraction layer and MLP to aggregate global information and predict whether the input sequence is valid (temporally coherent) or not.
3.4 ADAPTIVE UPSAMPLING
In most of the learning-based point cloud upsampling methods, all the points in the input will be upsampled and the ratio between the number of input points and output points is fixed. Given an upsampling network with upsampling ratio of r, for the point x; in the input point cloud, there will be r points in the upsampled point cloud that is directly upsampled from the input point z;. Since these networks are trained and operates on local patches, the locality will drive the upsampled points to be very close to the original input point. Every input point will have neighbors very close to them in the upsampled point cloud. However, not all point clouds are distributed a uniform way such that every point has similar amount of points in their neighborhood. For instance, in particle-based fluid dynamics, near-surface or flying away particles will have very different density compared to particles in the center of fluid field.
To suppress unnecessary upsampling, we propose learnable masking module, which predicts mask M that tells the network if needs to upsample a point. Concretely, for each point in the input point cloud, the masking module will predict confidence value Pt, which forms the soft mask vector
4

---

Published as a conference paper at ICLR 2022
Mp [p1: P2...., :. The soft mask is then quantized and transformed into binary mask Mr based on predefined threshold
PN
1. MMbe = 0,
[M 2 [M <E
(3)
In practice we set this threshold value to a small number (e.g. = 0.01*) so that the upsampling of a point will be suppressed only when the confidence level is low. Finally, use the binary mask to modulate the predicted residuals by doing an element-wii twiee multiplication, so that the residuals of unnecessary upsampling will be masked to zero. Note that here we use the binary mask modulate the output residuals instead of the input encoding of the upsampling module. The reason is that if we mask the features of some points to zero, then the upsampling module cannot aggregate features from these points when predicting the residuals.
During training, we generate the label M, for masking module by counting the number of neighbors of input points in the target point cloud Q
1,
[V(i) [W(I) <
[M]
0,
where the neighbor set: (1)) = {j ||, - r:112 < ro.j E Q} is derived by querying the neighbor of point in the target point cloud Q. The search radius ro is set be slightly larger than the average nearest distance between points the target point cloud. Based on the generated labels, we use L1 norm evaluate the loss between soft mask Mp and ground truth My:
Liassk N Σ
II[ [Mellll
(5)
3.5 TRAINING
Model architectures We follow the design of PU-GCN to build the feature extractor in the gen- erator, where we first Inception DenseGCN to extract features from the input points. Inside the upsampling and the masking module, we use two layers of GCN to aggregate features from neighbor- hood and use MLP to project the latent encoding into prediction. As for discriminators, we adopt the hierarchical structure similar the PointNet++, where we use set abstraction layer/flow embedding layer to extract features at different scales and continue to downsample the point cloud through network layers. (Model details can found Appendix A.1)
Loss function We adopt the least-squared GAN (Mao et al., 2017) setting to train model. Based on LS-GAN's formulation, we can derive the adversarial loss for temporal discriminator D, and spatial discriminator D.,
CD. = 111-1110 D. --11 +
(6)
Lp. DD))) D.
where X denotes the low-resolution input, Y denotes the high-resolution target, Y,, V, denotes three consecutive frames of point clouds: V, = (Y,1,, 11-1,,,, Yz, 1+ V. (G(X,-1), G(X.), G(X+1))} The corresponding adversarial losses for generator are:
LG = [] D,
(8)
Ca.s = - (.(())))
To encourage the output of generator lies close to the ground truth point clouds, we use Chamfer Distance (CD) to evaluate and penalize the distance between generator output Y and ground truth Y
LCD
1
Σ BEY
min 11 WEY
v1l +
1 Y
Σ
min ll 1
(10)
DEY BEY
'This threshold can be loosely (e.g. 0.1) without influencing final results much, since in practice most the mask values are very close or 0.0 after training.
IIf an input point can only find 0-2 neighbors in the ground truth point cloud, then we will mark this point as "no need upsample", upsampling it likely to introduce redundant points and violates point distribution.
5

---

Published as a conference paper at ICLR 2022
where y denotes point in Y, ij denotes point in Y. and denotes the number of points in the corresponding point cloud. Compared to other advanced distance metrics, Chamfer Distance is simpler and thus much faster to evaluate.
Combining with the masking loss discussed in the previous section, the total loss for generator is defined as:
LG La,t + La,s + ALD + X2
(11)
In practice we choose A1, 22 such that all loss terms have similar magnitude (roughly 0~~1).
Stabilizing GAN's Training We implement two techniques stabilize the training process. The first one is spectral normalization proposed by Miyato et al. (2018), which constrains the Lipschitz constant of the discriminator by setting the spectral norm of weights 1. We found this technique is crucial especially when training on the fluid dataset. We hypothesize the main reason is that the irregular distribution and complex dynamics of fluid particles can make discriminator prone to produce large gradients and thus make the training unstable. The second technique we adopt is label smoothing from Salimans et al. (2016), which softens the target label in the adversarial loss.
EXPERIMENT
4.1
IMPLEMENTATION DETAILS
Dataset We train and evaluate model two different datasets. The first dataset is generated from particle- based fluid dynamics simulation, where particles distribute and move under the physics governing equation. We use DFSPH (Bender & Koschier, 2015) solver from SPlisHSPlasHt and geometric assets from Ummenhofer et al. (2020) to generate the training data. We set the upsampling ratio to be r on the fluid dataset. The second dataset from scanned human action the MSR-Action dataset (Li et al., 2010), which consists of 567 Kinect depth videos of 20 action categories. this dataset, points from two different frames do not necessarily have correspondence and number of points varies in different frames. We adopt the same preprocessing and splitting strategy from previous works (Liu et al., 2020; Fan et al., 2021b). The upsampling ratio is set to be = 16 on the action dataset.
Evaluation metries In addition to the Chamfer distance discussed in the prvious section, we introduce three other spatial distance metrics further evaluate the model performance. The first metric is Earth Mover's Distance (EMD), which has been reported by previous work are more reliable and discriminative than Chamfer Distance dense point clouds' evaluation (Liu al., 2019a). EMD evaluates the distance between two clouds by solving linear assignment problem. Other than EMD, we use Maximum Mean Discrepancy (MMD) to evaluate the spatial distribution difference between prediction and ground truth. MMD discriminates the point clouds by measuring their high-level similarity instead of difference within pairs of matched points. In practice we found MMD is most discriminative on dense point clouds, thus we only report MMD the fluid dataset. We also report the Hausdorff distance (HD) following previous works in static point cloud upsampling task. The evaluation of the temporal coherence depending on the specific tasks, so we will discuss them in the corresponding sub-sections.
Baseline To demonstrate the effectiveness our proposed methods, we compare our model against two recent state-of-thh-art works, PU-GCN (Qian et al., 2021) and Tranquil Clouds (Prantl et al., 2020). Tranquil clouds proposes the first learning-basee method for upsampling point clouds in temporally coherent manner. Note that Tranquil Clouds uses mingling loss to encourage the points to spread and expand, this results in its much higher Chamfer Distance during evaluation, since Chamfer Distance favors upsampled points lying closer their original input points. The second baseline PU-GCN is the state-of-the-ar model on static point cloud upsampling.
(More details on metrics, baseline, and dataset generation be found in Appendix A.2, A.3, A.4)
Phtpps://ee thub. com/ Interacti vww.mptte http://wee...teeeeeeeeeerrrrrrrn /s/3/////11 1 1aHSP1as

---

Published as a conference paper at ICLR 2022
Fluids
Action
Input
TPU-GAN
Tranquil Clouds
PU-GCN
Figure 2:
Qualitative comparison of TPU-GAN against other models on different datasets. To fully evaluate the temporal coherence of upsampled point cloud sequences, we refer readers to the rendered videos, which are available at: [Videos].
4.2 DENSE AND NON-UNIFORM POINT CLOUDS FROM PHYSICAL SIMULATION
Advection consistency In fluids, particles' motion are subject to the underlying governing equation, such that given the current position and v.. there exists an continuous advection function: X++1 = (X,, v.) that advects particles from current state to next time step. Therefore we can evaluate the temporal coherence of generated sequences by checking if they are consistent with the underlying advection function. Given the ground truth data, we can use the kernel model in Smooth Particle Hydrodynamicss (SPH) (Gingold & Monaghan, 1977) to approximate the value of advection function at arbitrary locations (for more details on SPH and kernel model, please refer to the Appendix A.5). Based the approximated advection function v and the generator G, we define the advection consistency for generator as:
Cady = D (G(X V (G(X,), Va))
(12)
where X,, V denotes the positions and velocities particles at time step and D(P, Q) denotes the distance between two point clouds P, Q, which can be evaluated using the aforementioned spatial distance metrics.
Static
Advection Consistency
Model
CDJ x x
EMD| x10-2
MMDJ x10-5
HD |
CDL
EMDJ
MMDJ x10-5
HD +
x10-2
x10-4
x10-2
x10-2
TPU-GAN TPU-GAN + vel
2.856
2.694
5.373
3.978
1,963
1.582
5.069
5.835
2.964
2.722
5,058
4.009
1.632
1,434
3.545
5,593
PU-GCN 2.733
2.933
6.077
3.942
2.238
1.823
6.222
5,878
Tranquil Clouds
5.713
3.088
7.293
5.793
1.999
1.157
4.347
6.473
Table 1: Quantitative comparison of model performance on Fluids dataset. The Static column reports the per-frame difference between upsampled point clouds and ground truth. The Advection Consistency column reports the temporal coherence of consecutive frames of upsampled point clouds.
7

---

Published as a conference paper at ICLR 2022
Comparison with other models We tested out two versions of the proposed model with without using velocities as imput features. The quantitative evaluation results are shown in the Table I and the visualization of sequence upsampled from real coarse fluid simulation are shown in the left column of Figure 2. Compared to PU-GCN which does not take temporal coherence into account, TPU-GAN improves the advection consistency without the information of displacements across frames, which demonstrates the effectiveness of temporal discriminator. TPU-GAN also outperforms PU-GCN on high-level distance metries (EMD, MMD), which highlights the superiority of our proposed method in terms of reconstruction quality on irregular point clouds. Furthermore, we showcase that with velocity features, TPU-GAN is on par with Tranquil Clouds in terms of temporal coherence, while improves the static upsampling quality all the metrics. Note that Tranquil Clouds strongest when using EMD to evaluate advection consistency, in part because it trained with a temporal loss that is essentially equivalent to evaluating the advection consistency using EMD, with advection function approximated by EMD assignment instead of SPH's kernel interpolation.
4.3
REAL-WORLD SCANNED POINT CLOUDS
Frame
Sequence
Accuracy (%)
Model
Model
# Frames
CDL x10
EMD
HD 1
x10-2
10-
PT...
PointNet++
1
61.75
TPU-GAN
1.231
5.720
1.500
0.145
PSTNet
3
77.78
SA & MI.P
3
42.39
PU-GCN
1.173
5.884
1.503
0.156
Tranquil Clouds (8x)
2.262
5,987
1.669
0,201
TempoDis Feature
3
74.07
Tranquil Clouds (16x)*
2,483
6,243
1.697
0.224
+ SA & MLP
Table 2: Quantitative comparison of different models on
Table 3: MSR-Action 3D classification results
MSR-Action3D dataset.
using Temporal Discriminator's 's feature.
Perceptual loss for point cloud sequences For real-world scanned data, often don't know the corresponding points in the next frame given the current frame. Therefore it is difficult to directly compare the motion of upsampled sequence and target sequence. Inspired from previous video generation works which use pretrained I3D network (Carreira & Zisserman, 2018; Unterthiner al., 2018) to evaluate the spatio-temporal quality of generated videos at sequence level, we use a pretrained point cloud classifier to extract sequence-level classifier we used here is PSTNet (Fan et al., 2021b). the of model on point cloud sequence classification. We use the feature map from the middle of the network as the extracted features (more details can be found in Appendix A.3), and based the on the extracted features we define the perceptual loss (PL) for point cloud sequences as:
features. The pretrained
CpL. = |0 of xi)l
(13)
where Y = (Y1, predicted sequence and o denotes the feature extractor.
Fe,..., ..... Yr denotes the YT} denotes the reference sequence, Y = (81,8,,
Comparison with other models The quantitative evaluation result on MSR-Action 3D dataset is shown in the Table and visualization of upsampled point clouds is shown on the right column of the Figure 2. As indicated in the quantitative result, TPU-GAN can improve the upsampling quality in both frame-level and sequence-level under a scenario that no point correspondence annotation is available.
Self-supervised temporal feature learning To further evaluate the representation learned by tem- poral discriminator, we apply it as feature extractor for classification task. We freeze all the layers before the last set abstraction layer (SA) and MLP, then test the network's ' performance on action classification. In Table 3, we report the video-level accuracy for three supervised methods and classifier based on feature extracted by temporal discriminator. By leveraging spatiotemporal feature extracted by the temporal discriminator, the classifier outperforms PointNet++ that only utilizes dingle-framt features and outperforms model which uses only the set abstraction layer and MLP. PSTNet still leads the performance, it is trained in a fully-supervised way (i.e. all parameters trained on the classification task) and contains more parameters (PSTNet: 7.4M, TempoDis: 0.88M).
WWe tested two version of Tranquil Clouds 8x upsampling version (with 256 input points) following the original paper, and 16x upsampling version (with 128 input points).
8

---

Published as a conference paper at ICLR 2022
Table 4: Ablation on model architecture
Figure 3: Comparison of distribution of
points' top 0. % furthest distance
Frame
Sequence
Model
CDJ x x10
FMDL
HD J
12
Cround Truth with Spotial Dis weee Scatiai
PLJ
x10
x10
Full model
1,231
5.720
1.500
0.145
w/o SiatialDis
1.356
6.021
1.508
0.156
w/o TempoDis
1.185
5.821
1.458
0.155
Input two frames
1.201
5.743
1.489
0.149
Input five frames
1.295
5.946
1.535
0.159
4.4 ABLATION STUDY
Architectural choice We investigate the influence of our architectural choice the model per- formance on the MSR-Action 3D dataset. The result is shown in the Table 4. The result indicates that: first, increasing the frame number of point clouds exposed to the temporal discriminator can improve the the sequence level accuracy. However too long input sequence (" ("Input five frames") ") will deteriorate the frame-level upsampled quality. As it will make points cluster around input points and exhibit halo artifacts. (See Figure 10 in the Appendix for visualization)
Moreover, we find that spatial discriminator is important for the non- uniform dense point cloud upsampling. The upsampled points will shrink
Ground truth
P(a)
600
Mace 20j 5.3 Moow
towards the center of the point cloud without spatial discriminator. We calculates the top 0.1% furthest distances between every pair of points in the upsampled patches from fluid dataset, and we normalize all the distances using the furthest distance in ground truth. As shown in the Figure 3. without spatial discriminator, the upsampled point clouds' furthest distance distribution will shift the left.
Particie density n
TPU-GAN
Adaptive upsampling We analyze the point distribution in the up- sampled results from different methods by comparing their density to the ground truth. The analysis particle density is a crucial physical quantity that dominates the dynam- ics of incompressible fluids. We evaluate the partiele number density following the definition in the SPH model, and the result is shown in the Figure 4. For ground truth data, the density distribution has sharp peaks, which corresponds to particles near surface area and particles in the central part respectively. For upsampled point clouds, all the distri- bution has smoothed bell shape, while our method produce a sharper density distribution with a mode 7) = 7.4 more close to the ground truth (7) 6.3, 72 8.4). This demonstrates the adaptive masking module in the generator effectively learns to suppress unnecessary upsampling and approximates the target density distribution, while other methods have the density distribution shifted to the right.
100
conducted the fluid data, where
400
300
10
PU-GCN
500
Modein
400
300
200
100
Tranquil Clouds
Model size In Table of Appendix, we report the number of trainable parameters for concurrent tearning-based upsampling models. Despite the dual discriminator setting, our model is still much smaller than PU- Net and Tranquil Clouds which in turn makes it more efficient during inference.
Modern 9.6
500
400
200
100
CONCLUSION
Figure 4: Quantitative com-
In this work, we propose TPU-GAN, a GAN framework for upsampling dynamic point cloud sequences. Our proposed method is able to learn useful spatiotemporal representation for both generative task and discrim- inative task from raw point cloud sequences. Through experiments different data sources, we demonstrate the effectiveness of our method on the generative modelling of point cloud sequences.
parison of density distribu- tion of different upsampling methods on Fluid dataset.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work is supported by the start-up fund provided by CMU Mechanical Engineering.
REFERENCES
Jan Bender and Dan Koschier. Divergence-free smoothed particle hydrodynamics. In Proceedings of the 2015 ACM SIGGRAAPI/EEworrapiic Symposium on Computer Animation. ACM, 2015. doi: http://......ooo///1111477777722
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset, 2018.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks, 2019.
Mengyu Chu, You Xie, Jonas Mayer, Laura Leal-Taixé, and Nils Thuerey. Learning temporal coherence via self-supeo for gan-based video generation. ACM Transactions on Graphics, 39 (4), Jul 2020. ISSN 1557-7368. doi: 10. 10.1145/3386556..3324577 URL http:///d... doi. org/ 10. 11 45 5/338533366 3392457.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation complex datasets, 2019.
Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description, 2016.
Chao Dong, Chen Change Loy. Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks, 2015.
Hehe Fan, Yi Yang, and Mohan Kankanhalli. Point 4d transformer networks for spatio-temporal modeling in point cloud videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14204-14211 June 2021a.
Hehe Fan, Xin Yu, Yuhang Ding, Yi Yang. and Mohan Kankanhalli. PSTNet: Point spatio-tempora convolution point cloud sequences. In International Conference Learning Representations, 2021b. URL https: /ww.neveeeeeeeeeeeeeeeeeeeeeeeee net / forum? id=O3bgkf_Puya
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition, 2016
R. A. Gingold and J. J. Monaghan. Smoothed particle hydrodynamics: theory and application to non-spherical stars. Monthly Notices of the Royal Astronomical Society, 181(3)375-389, 12 1977. ISSN 0035-8711. doi: 10. 1..99//mra///81.3.2110 URL mnras /1cas/11111 3.555.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. Recurrent back projeprojoction network for video super-resolution, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham. Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using a generative adversarial network, 2017.
Ruihui Li, Xianzhi Li, Chi- ChigWing Fu, Daniel Cohen-Or, and Pheng-Ani Ann Heng. Pu-gan: point cloud upsampling adversarial network. In IEEE International Conference on Computer Vision (ICCV), 2019.
10

---

Published as a conference paper at ICLR 2022
Wanqing Li, Zhengyou Zhang, and Zicheng Liu. Action recognition based bag of 3d points. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 9-14, 2010. doi: 10. 10.100/CVPR22200054427..
Yangyan Li, Rui Bu, Mingehao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on X-trranfformed points, 2018.
Minghua Liu, Lu Sheng. Sheng Yang. Jing Shao, and Shi-Min Hu. Morphing and sampling network for dense point cloud completion, 2019a.
Xingyu Liu, Charles R. Qi, and Leonidas J. Guibas. Flownet3d: Learning scene flow in 3d point clouds, 2019b.
Xingyu Liu, Mengyuan Yan, and Jeannette Bohg. Meteornet: Deep learning dynamic 3d point cloud sequences, 2020.
Fan Lu, Guang Chen, Sanqing Qu, Zhijun Li, Yinlong Liu, and Alois Knoll. Pointinet: Point cloud frame interpolation network. Proceedings of the AAAI Conference on Artificial Intelligence, 35(3): 2251-2255 May 2021. URL https://ojos aaai www//iinde.e/eo..oo.oe/..oeoeoee php/ /AA.I/AAIa....ia. view/16324.
Wenjie Luo, Bin Yang, and Raquel Urtasun. Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net, 2020.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley, Least squares generative adversarial networks, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks, 2018.
Joe Yue-Hei Ng. Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification, 2015.
Lukas Prantl, Nuttapong Chentanez, Stefan Jeschke, and Nils Thuerey. Tranquil clouds: Neural networks for learning temporally coherent features in point clouds. In International Confer- ence on Learning Representations, 2020. URL https::tp/// /ttp:///ppeeeeeeeeeeeeeeeeeeeeee www.:neeef.ffiiiiiomommo BJeKh3VYDH.
Charles Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation, 2017 7a.
Charles R. Qi. Li Yi, Hao Su, and Leonidas J, Guibas, Pointnet++: Deep hierarchical feature learning on point sets in a metric space, 2017b.
Guocheng Qian, Abdulellah Abualshour, Guohao Li, Ali Thabet, and Bernard Ghanem. Pu-gen: Point cloud upsampling using graph convolutional networks, 2021.
Yue Qian, Junhui Hou, Sam Kwong, and Ying He. Pugeo-net: A geometry-centrie network for 3d point cloud upsampling, 2020.
Davis Rempe, Tolga Birdal, Yongheng Zhao, Zan Gojeic, Srinath Sridhar, and Leonidas J. Guibas. Caspr: Learning canonical spatiotemporal point cloud representations, 2020.
Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate
densely: Memory-efficient unsupervised training high-resolution temporal gan. International
May 2020. ISSN 1573-1405. doi: 10.1007) sl 1203--00--11333y. URL http: /tpp//x.dooo/ dol 1. orrg/1 110. 1007/ 11//33116666111 1111110/111222-0000013333 -20000033--------
Journal of Computer Vision, 128(10-11):2566-22
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, 2016.
Yiru Shen, Chen Feng, Yaoqing Yang, and Dong Tian. Mining point cloud local structures by kernel correlation and graph pooling, 2018.
11

---

Published as a conference paper at ICLR 2022
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks, 2015.
Sergey Tulyakov, Ming-Yo Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation, 2017.
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. In International Conference on Learning Representations, 2020.
http:///ppeeeeeeeeeeeeeee /
URL https:
net/ forum? ii=:ii
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A metric & challenges. arXiv preprint urXiv:1812.01717, 2018
Guangming Wang, Hanwen Liu, Muyao Chen, Yehui Yang, Zhe Liu, and Hesheng Wang. Anchor- based spatio-temporal attention 3-d convolutional networks for dynamic 3-d point cloud sequences. IEEE Transactions on Instrumentation and Measurement, 70:1-11, 2021a. ISSN 1557-9662. doi: 10.1109/tim.2221.30 1310100..
URL http:/ /xx doi. /t0.//0.0000 1109/ TIM. 2021. 3106101.
Kaisiyuan Wang. Lu Sheng, Shuhang Gu, and Dong Xu. Sequential point cloud upsampling by exploiting multi-scale temporal dependency. IEEE Transactions Circuits and Systems for Video Technology, pp. 1-1, 2021b. doi: 10.1109/TSSVT.2222111103333
Xintao Wang, Kelvin C. K. Chan, Ke Yu, Chao Dong, and Chen Change Loy. Edvr: Video restoration with enhanced deformable convolutional networks, 2019a.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph cnn for learning on point clouds, 2019b.
Huikai Wu, Junge Zhang, and Kaiqi Huang. Point cloud super resolution with adversarial residual graph networks, 2019.
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointcony: Deep convolutional networks on 3d point clouds, 2020.
You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. tempogan: temporally coherent, volumetric gan for super-resolution fluid flow. ACM Transactions on Graphics (TOG), 37(4):95, 2018.
Wang Yifan, Shihao Wu, Hui Huang, Daniel Cohen-Or, and Olga Sorkine-Hornung Patch-based progressive 3d point set upsampling. The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2019.
Lequan Yu, Xianzhi Li, Chi- Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. Ec-net: edge-aware point set consolidation network, 2018a.
Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Am Heng. Pu-net: Point cloud upsampling network, 2018b.
Yongheng Zhao, Tolga Birdal, Haowen Deng, and Federico Tombari. 3d point capsule networks, 2019.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 MODEL DETAILS
We implement model in PyTorch 1.7.1. All the training and experiments equiped with a single GTX-1180TI. We train all our model for 100k gradient updates using Adam optimizer (Kingma & Ba, 2017), which took approximately 30 hours on the Fluid dataset and 15 hours on the MSR-Action 3D dataset. The detail of our model architecture are listed below:
platform
For all the GCN, set abstraction and flow embedding layer, we use max pooling as aggregation function.
The feature extractor the Generator comprises one layer of GCN and two layers of Inception DenseGCN. At the GCN layer, we construct neighborhood by searching each points' 20 nearest neighbors based on the point position and use a MLP with hidden units: [3, 32,32, 128] to encode the point features. For Inception DenseGCN we follow the design and hyperparamter choice in the original PU-GCN paper (Qian et al., 2021).
The masking and upsampling module in the Generator has layer of GCN and [128, 64, cj MLP, where c = 3r for upsampling module, = for masking module and r is the upsampling ratio. The first layer of GCN operates on the 12 nearest neighbors and the second layer operates on the 4 nearest neighbors. Each GCN layer contains [128, 32] bottleneck linear layer and [32, 64, 128] ML.P to update point features.
We follow the design of PointNet++ to build the Spatial Dsicriminator. Concretely, use three set abstraction layers: (0.25., 32, [3, 64, 64, 128).. (0. (0.125N,,)) 32, [128, 128, 128)), (0.0625.N, 32, 1288, 128, 256)). (all, [256, 256, 512)) and [512, 256, 64, 1] MLP (with N the num- ber input points).
The Temporal Discriminator also adopts a hierarchical structure that contains two set abstraction layers: (0.25N,, 64, 33, 64, 64, 128)). (0.125N, 32, [128, 128, 256]). followed by two flow embed- ding layers:(0. 12,,, 32, [256, 256, 256]). (0.125 32, [256, 256, 256)), , and finally a set abstrac- tion layer: (all, [256, 256, 512)) and MLP: [512, 256, 64, 1).
Flow module
P
Flow Embedding layer
Flow Embedding layer
[3)
Output
Embedding layer
Embedding layer
F (0)
T-1 x shared layers
Flow Embedding Inyer
(0)
F
Flow Embedding layer
(0)
(T-2) x layers
Input
Figure 5: Schematic of flow module. The flow module takes frames of point cloud features
(F(") F(() F((") input. Inside each layer, consecutive frames of point clouds' features (F()) p()) are mixed via a shared flow embedding layer, then the output features F((+1) are stored on the former point cloud X,, For T input frames, the flow module will have depth of (T' different flow embedding layers).
$ (M, K. (......, with K sampled points, and then (hi....,h hi] MI.P update point features.
h:]) means that we Furthest Point Sampling (FPS) sample M local regions, each
13

---

Published as a conference paper at ICLR 2022
Feature extractor
Voctor concatenation
GCN
Inception DemseGCN
Inception DenieGCN
Input X
Output F
Masking / Upsampling module
GCN
GCN
MLP
Input F
Output
Figure 6: Top: Schematic of feature extractor in the generator. The feature extractor comprises three layers of Inception DenseGCN and concatenates features from different scales derive the final output. Bottom: Schematic of masking/upsampling module in the generator. Following the design of NodeShuffle module PU-GCN, inside masking/upsampling module, latent features are first aggregated from local neighborhood via GCN and then decoded into output by MLP.
Comparison on the number of parameters of different models is shown below.
Model
PU-Net
PU-GAN
PU-GCN
Tranquil Clouds
TPU-GAN
# Param. (M)
3.26
2.74
0.30
4.87
1.74
Table 5: Comparison model size of TPU-GAN vs other state-of-the-ant models
A.2 DATASET
Fluid We simulate 24 fluid sequences with random initialization of fluid block position and scene configuration, each sequence contains 200 snapshots with time interval of 0.005s and particle radius of 0.0125. Each snapshot contains roughly 100k 200k particles. We use 20 sequences as training data, and test on the rest 4 sequences. We adopt the patch training strategy, where we sample patches containing 8192 particles from the fluid field and uniformly downsample them with farthest point sampling create input-target pairs for training.
We follow the description from Liu et al. (2020) to process the raw data into point clouds. We uniformly downsample the processed data with farthest point sampling create low-resolution input and input the whole frame to the network. When evaluating metrics on MSR- Action 3D, normalize the point clouds to lie inside the unit ball [-1, 1] to balance the influence of point clouds of different classes. On MSR-Action 3D, we observe that masking module has less influence the final output and only very few points are masked.
MSR-Action 3D
A.3 METRIC
Earth mover distance We use Liu al. (2019a)'s EMD implementation" evaluate the loss. This implementation based on auction algorithm and thus is consistent with the EMD code used in the original Tranquil Clouds paper. In fact, our evaluation of Tranquil Clouds on the fluid dataset also roughly matches the results reported in their original paper (3.09 x 10 in our paper. 3.46 x 10 in original paper. the difference is likely caused by our fine-tuning on the pretrained weights and different implementation of EMD). To match the EMD metric reported in Prantl et al. (2020), convergence threshold of 0.03 is adopted on the Fluid dataset. For MSR-Action 3D, we use threshold of 0.002.
Maximum Mean Diserepancy We use the GeomLoss library to evaluate the MMD, with a Gaussian kernel and bandwidth o 0.01. The point clouds' coordinates are normalized to [0, 1] before evaluation.
7https: ///gi thub. httpbb.coo//olln////see// Pointtt Phttps://o thob.ccom//jeaffeeeyyggmmm
14

---

Published as a conference paper at ICLR 2022
Perceptual loss for point cloud sequence We evaluate the perceptual loss for action dataset based on features from the third convolution layer of PSTNet. As shown in the original paper (Fan et al., 2021b), the convolution layer output higher activation on the salient motion. This motivates to compare the activation of reconstructed point clouds against the ground truth, which provides a measurement for the similarity of two sequences in terms of motion. In practice we found that using activation from different middle layers (layer 2, 3, 4) produce similar results. To make the extracted feature permutation invariant to the input points, we conduct global top-k pooling (with 128) before calculating the loss.
A.4 BASELINE
Tranquil Clouds We use the publicly available source code" released by the authors to carry out experiment. On the fluid dataset, we find it unnecessary retrain the model since the fluid data in both papers are essentially generated from similar volume-conservin fine-tune the pretrained weights provided by the authors on our own fluid dataset for epochs. For action dataset, we explore two strategies to train the model. The first is to train the model from scratch without temporal loss, since there no point correspondence information in the dataset. However this makes Tranquil Clouds perform much worse in terms of temporal coherence. The second is directly using the pretrained weights which preserves temporally coherent latent space. general we found the latter provides much better temporal coherence thus we stick with pretrained weights to conduct experiment.
SPH simulator, so we just
We implement PU-GCN in PyTorch and compare our implemented version with the official TensorFlow version released by the authors We found they have similar performance. As our training and evaluation pipeline was mostly implemented in PyTorch, so for convenience we use PyTorch version of PU-GCN throughout the experiment.
PU-GCN
A.5 KERNEL MODEL IN SPH
common method to solve the Navier-Stokes equation and discretize fluids under the Lagrangian framework is Smooth Particle Hydrodynamics (SPH) (Gingold & Monaghan, 1977). In SPH, physical quantities at an arbitrary point in the space are approximated by the quantities of nearby particles.
Conretely, arbitrary scalar (or vector) field A (r) location r can be represented by convolution:
(r) = / A (r') W (Ir - ([r-rl1,1) 11) dV (r')
(14)
2 Σ
A(r;)
mi W ((r: -r,,,
(15)
Pj
JEN())
and p
Σ (E
m,W W (Ir; 1)),
(16)
where W is weighting function or smooth kernel as defined SPH, h is the cutoff, which defines the range of particles to be considered, V (r') denotes the volume Nii)) denotes the neighbor particles of particle that are within the cutoff, denotes the particle number density and m denotes
the mass of each fluid particle.
Using this model, we can approximate the advection & (r) at arbitrary location r by:
((r)
Σ JEN(()
(T, my W (Ir, - Pre--r,,Ah h)),
(17)
Pi
with
(t+1) (t) ((r,) = r
(18)
where r denotes the position of particle j at time step t. Throughout the experiment we use cubic
(t)
spline kernel function as the smooth kernel W()).
aabittabb.ooooo htteee////gtaaaacoo////ar///tttr "https: / // thub. coub.ccm//puuii http://cmmguucccegggga/a/oooo--
15

---

Published as a conference paper at ICLR 2022
A.6 VISUALIZATION AND STUDY ON THE MASKING MODULE
In this section we present visualization on the learnable masking module from different perspectives. Figure 7 shows the effect of masking module the very same network. Figure visualizes the prediction of masking module on randomly selected fluid patches. Figure 9 shows the superiority of using proposed masking loss (equation 4) train the masking module. Without the supervision of masking loss, the mask module will suppress the upsampling too much, which results in clustered points and halo structures.
Reference
Masking on
Masking off
Figure 7: Visualization of the effect of masking module on fluid dataset (zoom in for better view).
To "turn off" the masking module, we set all the values in the mask array predicted by the masking module to one.
16

---

Published as a conference paper at ICLR 2022
Input
Mask
Segmentation
Reference
Figure 8: Visualization of the masking module's s prediction (zoom in for better view). Given a sparse input, masking module will predict value for each point to indicate if it needs to be upsampled. Here we normalize all the mask value to [0, 1], and then color each point with their corresponding mask, red indicates mask value for this point close to zero (no need to upsample). blue indicates mask value is close to one (need to be upsampled).
Reference
Using masking loss
No masking loss
Figure 9: Visualization of the upsampling results based on different masking module training methods (zoom for better view). When not using masking loss, we remove the non-differentiable quantization process (cquation 3) and directly train the masking module based gradient from other losses (adversarial loss, Chamfer distance). Without masking loss' supervision, the upsampled results contain clustered points and halo structures.
17

---

Published as a conference paper at ICLR 2022
A.7 VISUALIZATION ABLATION
Figure 10: Visualization of snapshots from different sequences in MSR-Action 3D dataset, color indicates the depth of point. Top row: results from TPU-GAN with temporal discriminator taking 5 input frames; Bottom row: results from TPU-GAN with temporal discriminator taking 3 input frames. The results the top row exhibit halo structures.
A.8 VISUALIZATION POINT CLOUDS GENERATED DURING TRAINING PROCESS
10 nerations
20k iterations
100% iterations
Reference
Prediction
Visualization of fluid point cloud generated from model during the training process. The input and reference point clouds are randomly selected and cropped from the testing dataset. Point color indicates depth.
Figure 11:
18

---

Published as a conference paper at ICLR 2022
itcrations
20k iterations
100k iterations
Input
Reference
Prediction
Figure 12: Visualization of MSR-Action3D point cloud generated from model during the training process. The input and reference point clouds are randomly selected from the testing dataset. For better visualization, the input points shown in grey scale. Point color indicates depth.
A.9 SCENE-FLOW ESTIMATION
We apply proposed method to self-driving dataset (Kitti-scene flow) to demonstrate the effec- tiveness of our model in real-world self-driving application. We use the Kitti point cloud scene flow dataset proposed by Liu et al. (2019b). To mimic real world scenario where data very sparse, downsample the original point clouds (around 30000 40000 points per frame) to obtain sparse point clouds (2048 points per frame). We apply different upsampling networks upsample the sparse point clouds (visualization is shown in Figure 13). Then we use upsampled point clouds as input data for scene flow prediction network (pre-trained FlowNet3D (Liu al., 2019b)).
To quantitatively measure model's performance under different conditions, we adopt two metrics to evaluate the accuracy of predicted scene flow end point eπror (EPE 3D), which evaluates the L2 distance between ground truth and prediction, and 3D outliers, which evaluates how many predictions are extremely deviated from ground truth (>0.3m >5%). Since data very limited here (only 150 pairs of two consecutive frames are available), we follow previous works' evaluation protocol (Li et al., 2019; Qian et al., 2021; Liu et al., 2019b), all the upsampling networks are pretrained (on the fluid dataset) instead of training from scratch on Kitti scene data.
As shown in the Table 6, the performance of scene flow prediction network drops significantly when input points sparse, while upsampling networks can help mitigate the performance degrada- tion. Particularly. TPU-GAN outperforms other networks on both metries. This demonstrates the superiority of our proposed method and its potential self-driving application.
19

---

Published as a conference paper at ICLR 2022
Aatten is your
Reference points
Input 2048 points
TPU-GAN 16384 points
PU-GCN 16384 points
Tranquil Clouds
16384 points
Figure 13: Visualization Kitti-scene flow dataset. (Zoom in for better view)
Input source
EPE 3D4
3D outliers]
Full (16384 points)
0.1974
17.1%
Part (2048 points)
0.2577
27.5%
TPU-GAN 0.2243
19.8%
PU-GCN
0.2362
23.0 %
Tranquil Clouds
0.2392
22.5%
Table 6: Scene flow estimation performance of pre-trained FlowNet3D on different input sources.
Full means using all available points as input, following the setting in Liu et al. (2019h). Part means using the downsampled sparse data.
20


Reviews:
[{"review_id": "pCajtzNElH6", "reviewer": "Reviewer_waNz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the main issues of this paper are missing hyper-parameters experiments and unsatisfactory presentation quality. The authors are suggested to provide fair comparison under reasonable experimental settings, and improve the readability of the paper and further investigate the hyper-parameters to verify the robustness of the proposed method.\n\n\n-Methods\n- In Sec.3.1, the author mentioned only Inception DenseGCN is applied to the generator for efficiency. I am wondering how is the performance using the entire PU-GCN in the feature extractor? albeit the computational cost would increase.\n- In Sec.3.2, it is not clear that how the author obtain the $r_{j}^{t+1}$  according to $r_{i}^{t}$. Do you require mapping points in the current frame into the corresponding points in the next frame, or using the same position in two frames directly?\n- In Sec. 3.4, two parameters are introduced to the training process,(the predefined threshold $\\epsilon$ in equal.3 and the number ’3’ in equal.4 ), please justify your hyperparameter choice in the paper. \n- In Sec.3.5, the CD loss is used to evaluate the distance between generator output and ground truth due to its efficiency. How about using other losses? Could it bring huge computational costs?\n- In Sec.3.5, equal.6 and equal.12 have the same symbol $L_G$, which may cause confusion, It seems an adversarial loss was missed in your paper, please clarify.\n\n\nExperiments\n1. Missing ablation:\n- Explore up-sampling strategy: For up-sampling points, equal.5 requires the points to satisfy two conditions simultaneously. How about using equal.3 or equal.4 for up-sampling solely?\n- Each component of equal.13 shall be studied to verify the effectiveness of each block.\n2. Add visualization results: It is interesting that this work adopts a GAN to achieve super-resolution of points cloud. For this block, It would be better and convincing if the input and output point cloud during the training stage (for example, the first epoch and the final epoch) could be provided.\n", "summary_of_the_paper": "This paper proposes a framework called Temporal Point cloud Up-sampling to capture temporal and spatial features, in light of the difficulty to obtain point correspondence annotation. The framework is constituted by a generator for refining coarse point cloud, a temporal, and a spatial discriminator for feature extracting. Experiments conducted on two datasets towards human action recognition and fluid particles analysis demonstrate the effectiveness of this method.", "summary_of_the_review": "As mentioned in the main review, I am inclined to weakly reject this paper due to the missing hyper-parameters experiments, unsatisfactory presentation quality. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oh5dh1LZtMe", "reviewer": "Reviewer_gFvn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. Avoiding explicit supervision on the point displacements (scene flows)  is an important and big challenge for point cloud sequence modelling.  The direction that spatial-temporally upsamples point cloud sequences without scene flow supervision is correct and promising. \n\n2. The proposed method that employs a GAN framework for the super-resolution of dynamic point cloud sequence is technically sound.\n\n3. The proposed method is evaluated on two very different datasets and experiments demonstrates the effectiveness of the proposed methods. \n\nWeaknesses:\n\n1. Although addressing an interesting problem, the novelty and contribution seem a bit limited. GAN frameworks, such as PU-GAN,  have been used for point cloud upsampling. Modules, such as Set Abstraction (PointNet++) and Flow Module (FlowNet3D), are also widely used for static point cloud and point cloud sequence processing.  Combining these techniques to solve a new problem seems not that novel. \n\n2. The learnable masking module, which adaptively upsamples points, is a difference from previous works. It cloud be better to provide the visualization for this module to show what it learns. \n\n3. Self-driving is a more important application for point clouds. It cloud be better to apply the proposed method on self-driving car datasets to demonstrate its effectiveness. \n\n4. Upsampling three frames is somewhat short. It could be interesting to investigate the influence of sequence length on this task. \n\n5. A few related works are missing. For example, [a] interpolates frames into point cloud sequences, which can be seen as temporally upsampling.  [b] and [c] also learn Spatio-temporal representations of point cloud sequences without point displacement supervision. \n\n[a] PointINet: Point Cloud Frame Interpolation Network (AAAI21)\n\n[b] Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos (CVPR21)\n\n[c] Anchor-Based Spatio-Temporal Attention 3-D Convolutional Networks for Dynamic 3-D Point Cloud Sequences (TIP21)", "summary_of_the_paper": "This paper proposes a GAN framework for super-resolution on the dynamic point cloud sequences. \nThe main purpose is to avoid explicit supervision on the point displacements (scene flows) for this task.\nSpecifically, the Generator aims to produce super-resolution points for each individual frame. Then, a Temporal Discriminator and a Spatial Discriminator are used to measure the confidence of temporally coherent and spatially coherent, respectively. \nExperiments on fluid particles and 3D scanned human action that the proposed upsampling model can produce spatially and temporally point cloud sequences consistent with target data without explicit supervision on the point displacements.", "summary_of_the_review": "The paper presents an interesting problem. The proposed method is effective. However, the novelty seems a bit limited. Some experiments and related works are missing. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NlHizgOUdux", "reviewer": "Reviewer_evrR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths \n- TPU-GAN can learn temporal information from sequential point clouds and the experiments show that TPU-GAN can learn the underlying temporal coherence. \n- TPU-GAN can determine for each point whether it should be upsampled or not, and perform upsampling only for the points that need upsampling.\n\nweaknesses \n- There is no ablation study to confirm whether the function to determine for each point whether it should be upsampled or not is realized by the masking module. \n- The reason for employing the spatial discriminator should be explained.  I could not understand the reason why calculating temporal discriminator loss and spatial discriminator loss because temporal discriminator seems to include spatial discriminator in its structure.\n\nComment \n- The explanations of (b) and (c) in Figure 1 is reversed.\n- I think \"Table 4.3\" in the fourth line of the subsection \"Self-supervised temporal feature learning\" in section 4.2 is a mistake for \"Table 3\".\n- In the fourth line of \"Self-supervised temporal feature learning\" in section 4.2 \"Table 4.3\" may be \"Table 3\".\n", "summary_of_the_paper": "This paper introduced a method of point cloud upsampling using GAN, which can learn the underlying temporal coherence from a point cloud sequence.  TPU-GAN can determine for each input point whether it should be upsampled or not by introducing a masking module into the conventional point clouds upsampling model. In addition, TPU-GAN can learn the underlying temporal coherence using temporal discriminator that takes temporal sequential point clouds as input. The experiments are evaluated with particle-based fluid dynamics simulation dataset and scanned human action dataset. In the comparison,  the method proposed in this paper outperformed PU-GCN and Tranquil Clouds. Moreover, to confirm the performance variation with the number of input frames and to confirm the necessity of the both temporal discriminator and spatial discriminator in the learning, some other experiments are conducted. \n", "summary_of_the_review": "Please provide a short summary justifying your recommendation of the paper.  TPU-GAN is well designed to learn the underlying temporal coherence from a point cloud sequence, but additional experiments and additional explanations about the model and modules are necessary.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6Pdf6EYH4KP", "reviewer": "Reviewer_7Wbg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes TPU-GAN,  an approach for point cloud generation concerning temporal coherences. In addition, the authors introduce a new trainable mask for controlling the distribution of sampled points. Both methods are well motivated. The authors provide experiments investigating various aspects of the proposed approaches. While the results seem to be reasonable, there are several concerns stated below.\n\nMajor comments:\n\n1. I find methods to be well explained and experimental part to be sufficient. The authors' results are reasonably better than others analyzed methods. Nevertheless, I find that the main contribution, TPU-GAN, is based on several well-known ideas or techniques with small, in my opinion, novelty. Here is how I understand the novelty of the work: there already exist approaches that use GAN, for example, PU-GAN and PU-GCN. The novelty here is to use a GAN that incorporates learning temporal coherence. However, in my opinion, the background behind the TPU-GAN is based on existing approaches: feature extractor follows PU-GCN,  Temporal Embedding Layers used in the architecture are from work [4] with a small difference in input features ($ F_t, F_{t + 1}$). Spatial and temporal loss functions are common for GANs. In addition, the idea of using the triple moment($t - 1, t, t + 1$) for training the model appears in the paper [5].  Please, give clear arguments to show the significant novelty of TPU-GAN. \n\n2. It’s claimed in the text that “Tranquil Clouds requires point correspondence across frames to evaluate the first-order and second-order finite difference loss, which our method (TPU-GAN) does not require during the training phase.” However, TPU-GAN’s loss function still relies on ground truth in $L_{CD}$ . Please, clarify what exactly was meant by this statement? \n\nMinor comments:\n\n1. I recommend the authors look through the recent work \"Sequential point cloud upsampling by exploiting multi-scale temporal dependency\" by Wang K [3] and discuss the similarities and differences with the proposed TPU-GAN. In this paper, the authors also proposed an approach for non-uniform sparse upsampling with preserving temporal dependence. Although their results for MSR-Action 3D are a little bit lower in terms of Chamfer Distance, they don't estimate temporal coherence, and probably the TPU-GAN is significantly better in this metric. Of course, the authors could find even more advantages.\n2. Papers [1, 2, 3] consider Hausdorff Distance and Point-to-surface distance as metrics. Is it reasonable to include them in the analysis? \n3. In equation (2), what is $\\gamma$ and $h$? Are they similar to the same in the equation (1)? \n4. In equation (3), why is it used 3 as the threshold? \n5. Please, proofread the work. Examples: \nAbstract. “In this work…” - should be separated by a comma; \nSection 2. “Compared to PU-GAN…” - seems should be to PU-GCN or the reference on PU-GAN should be added;  \nSection 2. “...the change of points with respect to time are not taken into consideration” - use “is” instead of “are”;\n\nReferences:\n1. Li, Ruihui, et al. \"Pu-gan: a point cloud upsampling adversarial network.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n2. Qian, Yue, et al. \"Pugeo-net: A geometry-centric network for 3D point cloud upsampling.\" European Conference on Computer Vision. Springer, Cham, 2020.\n3. Wang, Kaisiyuan, et al. \"Sequential point cloud upsampling by exploiting multi-scale temporal dependency.\" IEEE Transactions on Circuits and Systems for Video Technology (2021).\n4. Liu, Xingyu, Charles R. Qi, and Leonidas J. Guibas. \"Flownet3d: Learning scene flow in 3d point clouds.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n5. Prantl L. et al. Tranquil clouds: Neural networks for learning temporally coherent features in point clouds //arXiv preprint arXiv:1907.05279. – 2019.\n", "summary_of_the_paper": "In the paper, the authors firstly introduced TPU-GAN, a GAN that provides a temporal coherence for point cloud sequence.  Another contribution is an upsampling module built into TPU-GAN. In contrast to other methods that produce almost uniform sampling, this probability module provides a more reasonable distribution of upsampling points.\nThe authors investigate proposed methods both quantitatively and qualitatively on two different datasets. They show that TPU-GAN performs better than other state-of-the-art methods in terms of static view (per-frame) and temporal coherence. They also prove that the upsampling model in TPU-GAN provides distribution closer to ground truth distribution.\n", "summary_of_the_review": "The proposed methods are well-investigated from different aspects in two different datasets. The paper is well-structured and well-written; most of the claims and hypotheses are supported by good arguments. However, it remains the question about the novelty's significance. If the authors provide reasonable explanations, I am ready to recommend the paper for acceptance.\n \nPlease, let me add a comment that I’m not an expert in the field of models for point clouds and admit that I could understand something wrong, but I've spent a lot of time reviewing the literature. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3tzHXYL7pqy", "reviewer": "Reviewer_dDgo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The problem addressed by the paper is significant. Leveraging temporal cues for point cloud upsampling is interesting and meaningful.\n\n2. The approach makes sense. Approaches that leverage scene flow is likely to add too much computational burden and the annotation is very expensive to obtain. Using a discriminator can be a useful alternative.\n\n3. The tasks are very interesting and commercially valuable. I am especially fond of the fluid mechnics simulation, and the potential commercial value behind it is immense.\n\n4. The paper is coherent and clear, it is easy to follow.\n\n5. Effective results in the visualization (Figure 2 and 3) where the proposed method clearly has a competitive edge. \n\n6. Suitable baselines are chosen in the experiments: PU-GCN (static SOTA) and Tranquil Clouds (first learning-based temporal method).\n\nWeaknesses:\n\n1. The method is largely a combination of existing works. The basic modules may take less space and have their own \"preliminaries\" section: set abstraction that is taken from PointNet++, multi-scale graph convolution from PU-GCN, and temporal feature extraction module flow embedding from Flownet3d. More importantly, the critical part of the nolvelty, the temporal and spatial discriminators have simplistic designs by putting the above-mentioned modules together. This is may be the first work to leverage temporal information with discriminators, having a straight-forward design is understandable, but adaptation to the given tasks may improve the relevance.\n\n2. The experiment setting may raise some questions, and the quantitive evaluation seems to have some performance gap. I understand that existing datasets for point cloud upsampling are all static, but I would like to hear the authors' comments on if it makes sense to tweak the conventional setting and introduce motion into the existing datasets (e.g. sample point clouds with a moving virtual camera or sample point clouds from a moving object, as these settings have some use cases in real life). Moreover, the performance of the proposed methods are not outperforming existing methods entirely (Table 1 and 2).\n\n\nMinor comments:\n\n1. Please elaborate on how duplicate points are removed in Figure 1a)\n\n2. Sec 3.3: \"often no correspondences\" -> \"often no clear correspondences\". As there are definately some correspondences but we are unable to know.\n\n3. Sec 3.3: \"... remains unknown\" -> \"are not annotated\"\n\n4. Sec 3.4: \"However, not all point clouds are distributed in a uniform way such that every point has similar amount of points in their neighborhood\": solid observation, but this is also the case for static point cloud upsampling. Does point cloud sequence makes this problem more prominent? I would assume adaptive upsampling is especially critical for point cloud sequences as it is highlighted in the abstract.\n\n5. Sec 3.5: lambda 1 and lambda 2 are different for different datasets, does this mean the hyperparameters are sensitive?\n\n6. Sec 4.1 Dataset: why do you use different upsampling ratio for two tasks?\n\n7. Would authors release the dataset for fluid simulation?\n\n8. Sec 4.2 Comparison with other models: it is claimed that TPU-GAN outperforms PU-GCN on \"high-level\" distance metrics, does this make CD not an effective metric?\n\n9. Page 8 footnote 2: input size is a critical factor, would it be possible to adapt the code of Tranquil Clouds to allow for 16x upsampling?\n\n10. Conclusion and Figure 4 captions are excessively close to each other.", "summary_of_the_paper": "This paper proposes to use a discriminator to enhance the temporal coherence for point cloud upsampling. The technique is tested in two scenarios: fluid mechanics simulation and human scan.", "summary_of_the_review": "The paper has clearly several weaknesses, but it is undeniable that it has brought forward an interesting topic of point cloud upsampling with temporal cues, and developed two relevant tasks that are useful in real life. I thus opt for \"marginally above the acceptance threshold\" as the initial rating and hope the authors could address my concerns during the rebuttal period.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a GAN framework for dynamic point cloud superresolution. It does not need scene flow supervision for training and has an interesting adaptive upsampling mechanism. Results are shown on several datasets and are reasonably convincing. Overall, all the reviewers are slightly positive about the work. After the rebuttal, all the five reviewers converged to a marginally-above-the-threshold recommendation. The meta-reviewer agreed with their assessment and would like to recommend accepting the paper.
Decision: Accept (Poster)