Tranquil Clouds: Neural Networks for Learning Temporally Coherent Features in Point Clouds
Authors: Lukas Prantl
Nuttapong Chentanez
Stefan Jeschke
Nils Thuerey
Venue:  (2020)
Keywords: point clouds
spatio-temporal representations
Lagrangian data
temporal coherence
super-resolution
denoising

Abstract: Point clouds, as a form of Lagrangian representation, allow for powerful and flexible applications in a large number of computational disciplines. We propose a novel deep-learning method to learn stable and temporally coherent feature spaces for points clouds that change over time. We identify a set of inherent problems with these approaches: without knowledge of the time dimension, the inferred solutions can exhibit strong flickering, and easy solutions to suppress this flickering can result in undesirable local minima that manifest themselves as halo structures. We propose a novel temporal loss function that takes into account higher time derivatives of the point positions, and encourages mingling, i.e., to prevent the aforementioned halos. We combine these techniques in a super-resolution method with a truncation approach to flexibly adapt the size of the generated positions. We show that our method works for large, deforming point sets from different sources to demonstrate the flexibility of our approach.

Published as a conference paper at ICLR 2020
TRANQUIL CLOUDS: NEURAL NETWORKS FOR LEARNING TEMPORALLY COHERENT FEATURES IN
POINT CLOUDS
Lukas Prantl
Nuttapong Chentanez
Department of Computer Science Technical University of Munich
NVIDIA
Bangkok, Thailand
Munich, Germany
Stefan Jeschke
Nils Thuerey
NVIDIA
Department of Computer Science Technical University of Munich
Vienna, Austria
Munich, Germany
ABSTRACT
Point clouds, as form of Lagrangian representation, allow for powerful and flexible applications in large number of computational disciplines. We propose a novel deep-Iearning method to learn stable and temporally coherent feature spaces for points clouds that change over time. We identify set of inherent problems with these approaches: without knowledge solutions exhibit strong flickering, and easy solutions to suppress this flickering can result undesirable local minima that manifest themselves as halo structures. We propose novel temporal loss function that takes into account higher time derivatives of the point positions, and encourages mingling, i.e., to prevent the aforementioned halos. We combine these techniques in a super-resolution method with a truncation approach to flexibly adapt the size of the generated positions. We show that our method works for large, deforming point sets from different sources to demonstrate the flexibility of our approach.
the time dimension, the inferred
INTRODUCTION
Deep learning methods have proven themselves as powerful computational tools in many disciplines, and within it topic of strongly growing interest is deep learning for point-based data sets. These Lagrangian representations are challenging for learning methods due their unordered nature, but are highly useful in a variety of settings from geometry processing and 3D scanning to physical simulations, and since the seminal work Qf Charles al. (2017). range of powerful inference tasks can be achieved based on point sets. Despite their success. interestingly. no works so far have taken into account time. Our world, and the objects within it. naturally move and change over time, and as such it is crucial for flexible point-based inference to take the time dimension into account. In this context, we propose method learn temporally stable representations for point-based data sets, and demonstrate its usefulness in the context of super-resolution.
An inherent difficulty point-based data is their lack of ordering, which makes operations such as convolutions, which are easy to perform for Eulerian data, unexpectedly difficult. Several powerful approaches for point-basee based convolutions have been proposed (Qi et al. 2017: Hermosilla et al. 2018, Hua et al. 22018), and we leverage similar neural network architectures in conjunction with the permutation inniraattiooniniiantt Earth Mover's Distance (EMD) to propose a first formulation of loss for temporal coherence.

---

Published as a conference paper at ICLR 2020
In addition, several works have recognized the importance of training point networks for localized patches, in order to avoid having the network to rely on a full view of the whole data-set for tasks that inherently local, such normal estimation (Qi Charles et al. 2017). and super-resolution (Yu et al. 2018a). This also makes it possible to flexibly process inputs of any size without being limited by memory requirements. Later we will demonstrate the importance of such a patch-based approach with sets of changing cardinality in our setting. A general challenge here is to deal with varying input sizes, and for super-resolution tasks, also varying output sizes. Thus, in summary we target an extremely challenging learning problem: we are facing permutation permutation-ivvaiiant inputs and targets of varying size, that dynamically move and deform over time. In order to enable deep learning approaches in this context, we make the following key contributions: Permutation invariant loss terms for temporally coherent point set generation; A Siamese training setup and generator architecture for point-based super-resolution with neural networks; Enabling improved output variance by allowing for dynamic adjustments of the output size; The identification of specialized form of mode collapse for temporal point networks, together with loss term to remove them. We demonstrate that these contributions together make it possible to infer stable solutions for dynamically moving point clouds with millions of points.
More formally, we show that our learn- ing approach can be used for generating a point set with an increased resolution from given set of input points. The gen- erated points should provide an improved discretization of the underlying ground truth shape represented by the initial set of points. For the increase, we will target a factor of two to three per spatial dimension. Thus, the network has the task to estimate the underlying shape, and to generate suit- able sampling positions as output. This is generally difficult due to the lack of connectivity and ordering, and in our case, positions that move over time in combination with changing number of input points. Hence is crucial that the network able to establish a temporally stable latent space representation. Although point at time t moved to new location at time & + At, the points can arbitrarily change their relative position and density over the course of their movement, leading to a substantially more difficult inference problem than for the static case.
a)
Figure I: Our algorithm upsamples an input point cloud (a) in temporally coherent manner. Three exemplary outputs are shown in yellow in (b).
assume that we know correspondences over time, i.e., we know which
RELATED WORK
Deep learning with static point sets was first targeted in PointNet (Qi Charles et al. 2017) via order-invariant networks, while PointNet++ (Qi et al. 2017) extended this concept to generate convolution operation for grid-based data. This concept
features for localized groups similar can be hierarchically applied to the generated groups, in order to extract increasingly abstract and global features. Afterwards, the extracted features can be interpolated back to the original point cloud. The goal to define point convolutions has been explored and extended in several works. The MCNN approach (Hermssilaa et 2018) phrased convolution in terms of Monte Carlo integration. PointCNN (Hua et al. 2018) defined a pointwise convolution operator using nearest neighbors, while extension-restrictin function were used in Atzmon et al. (2018). The PU-Net Yu et al. 2018a) proposed a network for upsampling point clouds, and proposed a similar hierarchical network structure of PointNets along the lines of PointNet++ to define convolutions. Being closer to our goals, we employ this approach for convolutional operations our networks below. We do not employ the edge-aware variant of the PU-Net (Yu et al. (0018) here, to keep it as simple and general as possible we focus temporal changes in our work.
operators for mapping between a point cloud function and a volumetric
Permutation invariance is central topic for point data, and was likewise targeted in other works (Ravanbakhsh et al.) 2016 Zaheer et al., 2017). The Deep Kd-network (Klokov and Lempitsky 2017) defined hierarchical convolution on point clouds via kd-trees. PointProNets (Roveri et al. 2018) employed deep learning to generate dense sets of points from sparse and noisy input points for 3D reconstruction applications. PCPNet (Guerrero al. 2018), as another multi-scale variant of
2

---

Published as a conference paper at ICLR 2020
PointNet, has demonstrated high accuracy for estimating local shape properties such normal or curvature. P2PNet (Yin et al.] 2018) used a bidirectional network and extends PointNet++ to learn a transformation between two point clouds with the same cardinality.
Recently, the area of point-based learning has seen a huge rise in interest. One focus here are 3D segmentation problems, where numerous improvements were proposed, e.g., by SPLATNet (Su et al.. 2018), SGPN (Wang et al. 2018a). SpiderCNN (Xu et al. 2018), PointConv (Wu et al. 2018), SO- NEt(Li al. 2018a) and 3DRNN Yee et al. 2018). Other networks such Flex Convolution (Groh) et al. 2018), 8 the SuperPoint Graph (Landrieu and Simonovsky 2018), and the fully convolutional network (Rethage et al., 2018) focused on large scale segmentation. Additional areas of interest are shape classification [Wang et al., 2018b; Lei et al.) [2018] Zhang and Rabbat 2018 Skouson 2018) and object detection (Simon et al. 2018 Zhou and Tuzel 2018), and hand pose tracking (Ge et al. 2018). Other works have targeted rotation and translation invariant inference (Thomas et al. 2018). and point cloud autoencoders (Yang et al. 2018 Deng et al. 2018). A few works have also targeted generative models based on points, e.g., for point cloud generation (Sun et al. 201 8), and with adversarial approaches (Li et al. 2018b). It is worth noting here that despite the huge interest, the works above do not take into account temporally changing data, which is the focus of our work. notable exception is an approach for scene flow (Liu et al., (2018), in order to estimate 3D motion directly on the basis of point clouds. This work is largely orthogonal to ours, as it does not target generative point-based models.
3 METHODOLOGY
We assume an input point cloud X = {I1, To, Fyk. of size k E [1, kmaz). consists of points I; E R. where d includes 3 spatial coordinates and optionally additional features. Our goal is to let the network (x) infer function Y which approximates desired super-resolution output point cloud Y {y1. 3/2, .... 3r of size [1, Mmaz] with y4 R3, i.e. fX) = Y se Y. For now assume that the number of output points n is defined by multiplying with a user-defined upsampling factor r, i.e.
rk. Figure 21) illustrates the data flow in our super-resolution network schematically. We treat the upsampling problem as a local one, i.e., we assume that the inference problem can be solved based on a spatially constrained neighborhood. This allows us work with individual patches extracted from input point clouds. At the same time, it makes it possible to upsample adaptively, for example, by limiting the inference to relevant areas, such as complex surface structures. For the patch extraction we use a fixed spatial radius and normalize point coordinates within each patch to lie in the range of [-1,1] 1].
Our first building block is measure for how well two point clouds represent the same object or scene by formulating an adequate spatial loss function. Following Achliiptas et al. (2017), we base our spatial loss Ls on the Earth Mover's Distance (EMD), which solves an assignment problem to obtain a differentiable bijective mapping Φ : 5 + y. With ø we can minimize differences in position for arbitrary orderings of the points clouds via:
Ls min
1 (
0:0 +V
3.1 TEMPORAL COHERENCE
a) k
b)
x
When not taking temporal coherence explicitly into account, the highly nonlinear and ill-posed nature of the super-resolution problem can cause strong variations in the output even for very sub- tle changes in the input. This results in sig- nificant temporal artifacts that manifest them- selves as flickering. In order to stabilize the output while at the same time keeping the net- work structure small and simple as possible, we propose the following training setup. Given sequence of high resolution point clouds Y*. with indicating time, we can compute a veloc- ity y = (uf, vo,,,,, v where of IR3 For
pod
pod
Fk
NN
[(aa)
f,00)
1.60)
Iraxe
Figure 2: a) Schematic overview of fx(X). Black ar- rows represent scalar data. Point data is depicted as colored arrows with the color indicating data cardinality (brown=k, red kmaz. green = Rmaax blue = n, and purple f). b) Siamese network setup for temporal loss calculation.
3

---

Published as a conference paper at ICLR 2020
this finite difference (v) (u]+1 yf), where assumee, without loss of generality, At 1, i.e. the time step is normalized to one. For training, the low resolution inputs X can now be generated from Y via down-sampling by factor of r, which yields subset of points with velocities. Details of our data generation process will be given below.
To train temporally coherent network with the Y sequences, we employ Siamese setup shown in Figure(p. We evaluate the network several times (3 times in practice) with the same set of weights, and moving inputs, in order enforce the output behave consistently. In this way we avoid recurrent architectures that would have to process the high resolution outputs multiple times. In addition, we can compute temporal derivatives from the input points, and use them to control the behavior of the generated output.
Under the assumption of slowly moving inputs, which theoretically could be ensured for training, straightforward way to enforce temporal coherence would be to minimize the movement of the generated positions over consecutive time steps terms of L2 norm:
L2v Σ 11 12.
(2)
While this reduces flickering. it does not constrain the change of velocities, i.c., the acceleration. This results in high frequency jittering of the generated point positions. The jitter can be reduced by also including the previous state at time step t 1 to constrain the acceleration in terms of its L2 norm:
L2A = Σ 10 - + 11
However, a central problem of a direct temporal constraint via Equations (2 and 3 consistently leads to highly undesirable clustering of generated points around the center point. This is caused by the fact that the training procedure as described so far is unbalanced, as it only encourages minimizing changes. The network cannot learn to reconstruct realistic, larger motions in this way, but rather can trivially minimize the loss by contracting all outputs to a single point. For this reason, we instead use the estimated velocity of the ground truth point cloud sequence with a forward difference in time, to provide the network with reference. By using the EMD-based mapping established for the spatial loss in Equation we can formulate the temporal constraint in permutation invariant manner
is that
LEV = Σ ( 51 - (
(4)
Intuitively, this means the generated outputs should mimic the motion of the closest ground truth points. As detailed for the L2-based approaches above, makes sense also take the ground truth acceleration into account to minimize rapid changes of velocity over time. We can likewise formulate this in a permutation invariant way w.r.t. ground truth points via:
LEA Σ (1
27 + Dt (+()) 20(0)) + Φ(T. 1)
(5)
We found that combination of LEV and LEA together with the spatial loss Ls from Eq. provides the best results, as we will demonstrate below. First, we will introduce the additional loss terms of algorithm.
3.2 VARIABLE POINT CLOUD SIZES
Existing network architectures are typically designed for processing fixed amount of input and output points. However, in many cases, and especially for a localized inference of super-resolution, the number of input and output points varies significantly. While we can safely assume that no patch exceeds the maximal number inputs kmaz (this can be ensured by working on a subset), it easily happen that certain spatial region has fewer points. Simply including more distant points could guarantee that we have a full set of samples, but this would mean the network has to be invariant scaling, and to produce features different spatial scales. Instead, train
4

---

Published as a conference paper at ICLR 2020
Figure 3: An illustration of the relationship between input and output size. (a, show histograms of point set sizes for: (a.b) the input set; (c) the ground truth target sets; and (d) the network output, ic. times larger than the imput. The latter deviates from the ground truth in (c). but follows its overall structure. This is confirmed in (b). which shows heat map visualization of input vs. ground truth output size. The diagonal structure of the peak confirms the approximately linear relationship.
301
Figure 4: The effect of our variable oulput handling for exemplary patches. In red the ground truth target, in blue the inferred solution. Left (a) with fixed output size, and the right (b) with the proposed support for variable output sizes. The latter approximates rather uniform shapes that, c.g., cover empty space above the ground truth in both examples.
shape of red ground truth points significantly better. (a) leads to
network for a fixed spatial size, and ensure that it can process varying numbers of inputs. For inputs with fewer than kmaz points, we pad the input vector to have a fixed size. Here, we ensure that the padding values are not misinterpreted by the network as being point data. Therefore, we pad X with p E {-2] which represents a value outside the regular patch coordinate range [-1, 1]: X
{2), 22 ..... The P.P...... The first convolutional layer in our network now filters out the
kmaz
padded entries using the following mask: Min = {mmee(o,a] = 1,, 1,0,0,.., The entries
Ammaz -
of p allow us to compute the mask on the fly throughout the whole network, without having to pass through For input of size k. our network has the task to generate ñ = rk points. As the size of the network output is constant with rkmen, the outputs are likewise masked with Mout to truncate it length for all loss calculations, e.g., the EMD mappings. Thus, as shown in Figure 2p is used to truncate the point cloud Y = {11, U2. Inmaz via mask Mout to form the final output Y -ūal / E [1,
Note that Sec. 3.J. we have for simplicity assumed that n = rk, however, in practice the number of ground truth points n varies. As such, ñ only provides an approximation of the true number of target points in the ground truth data. While the approximation is accurate for planar surfaces and volumes, it is less accurate in the presence of detailed surface structures that are smaller than the spatial frequency of the fow-resolution data.
We have analyzed the effect of this approximation in Fig. The histograms show that the strongly varying output counts are an important factor in practice, and Fig. adddiiionally shows the improve- ment in terms of target shape that results from incorporating variable output sizes. In general, ñ provides a good approximation for our data sets. However, as there a chance to infer an improved estimate of the correct output size based on the input points, have experimented with training second network to predict ñ in conjunction with a differentiable output masking. While this could be an interesting feature for future applications, we have not found to significantly improve results. As such, the evaluations and results below will use the analytic calculation, i.c., ñ = rk.
3.3 PREVENTING HALO ARTIFACTS
For each input point the network generates output points, which be seen individual groups g: (()) ( [rg 1, (r + 1) These groups of size in the output strongly related to the input points they originate from. Networks that focus on maintaining temporal coherence for the dynamically changing output tend to slide into local minima where r output points attached
5

---

Published as a conference paper at ICLR 2020
Waboty
Figure 5: Ablation study for temporal loss formulation. Black points indicate targets, while green points are generated (both shown as time average). a) Result from previous work: b) With L2v loss; e) the proposed velocity loss Lgv; d) our full loss formulation with LEv + LEAA While (a) has difficulties approximating the target shape and the flickering output is visible as blurred positions, the additional loss terms (esp. in (c) and (d)) provide stable results that closely approximate the targets. Note that (b) leads to undesirably static motion near the bottom of the patch. As the input points here are moving the output should mimic this motion, like (c,d).
fixed structure to the input point location. This manifests itself as visible static halo-like structures that move along with the input. Although temporal coherence is good in this case, these cluster-like structures lead to gaps and suboptimal point distributions in the output, particularly over time. These structures can be seen as form of temporal mode collapse that can be observed in other areas of deep learning, such as GANs. To counteract this effect, we introduce an additional mingling loss term to prevent the formation of clusters by pushing the individual points of a group apart:
|+( Σ +
LM
(6)
#
o, Ex())
Note that in contrast to previously used re- pulsion losses (Yu et al. 2018a), Las en- courages points globally mix rather than just locally repelling each other. While a repulsion term can lead deterioration of the generated outputs, our formulation preserves spatial structure and temporal co- herence while leading to well distributed points, as illustrated in Fig. 6
Figure Left, result without the mingling loss from Eq. right with (a single point group highlighted in orange). The former has many repeated copies of a single pattern, which the mingling loss manages to distribute as can be seen in the right picture.
In combination with the spatial and temporal terms from above, this leads to our final loss function finiall Ls + 7LEV + pLEA + vLas, with weighting terms H.
EVALUATION AND RESULTS
We train our network fully supervised manner with simulated data, To illustrate the effect of our temporal loss functions, we employ it in conjunction with established network architectures from previous work (Qi Charles et al. 2017; Yu et al., 2018a). Details of the data generation and network architectures are given in the appendix. We first discuss data generation and training setup, then illustrate the effects of the different terms of our loss function, before showing results for more complex 3D data sets. As results focus temporal coherence, which is best seen in motion, we refer readers to the supplemental materials at https:: /ge. in, tum, de/ wttppubpliieeee powbliccaiion 2020- -clr prant order fully evaluate the resulting quality.
Ablation Study We evaluate the effectiveness of our loss formulation with two dimensional ablation study. An exemplary patch of this study is shown in Fig. 5 In order to compare our method to previous work, we have trained previously proposed method for point-cloud super-resolution, the PU-Net (Yu et al. 201 Sa) which internally uses a PointNet++ (Qi et al., 2017). with our data set, the only difference being that we use zero- pero-padding here. This architecture will be used in the following comparisons with previous work. Fig. 55) shows a result generated with this network. As this figure contains average of multiple frames to indicate temporal stability, the blurred regions, esp. visible on the right side of Fig. Fig.5h), indicate erroneous motions in the output. For this network the difficulties of temporally changing data and varying output sizes additionally lead to suboptimal approximation

---

Published as a conference paper at ICLR 2020
Ls
LN
LM
Lav
LEV
LEA
2D Previous work
0.0784
0.329
5.499
0.1
0.402
0.107
0.214
2D With Lav
0.044
0.00114
2.197
1.Ie-05
4.2c-05
0.00197
0.00276
2D Only LEv
0.0453
0.00114
2.713
2.6c-05
6.0e-06
6.15e-04
5.27c-04
2D Full
0.0487
0,00116
3.0307
2.1e-05
1,0e-06
6,52e-04
1.46e-04
3D Previous work
0.0948
0.494
10,558
0.325
1.299
0.19
0.365
3D Full
0.0346
0.00406
3.848
8.04c-04
2.0e-06
0.00179
7.09e-04
Table 1: Quantitative results for the different terms of our loss functions, first for our 2D ablation study and then for our 3D versions. The first three columns contain spatial, the next four temporal metrics. LN = || nll2 is given measure of accuracy in terms of the size of the generated outputs (it is not part of the training).
Figure 7: Illustrations of the latent spaces learned by our networks. (a) shows averaged latent space values for 100 random patch sequences of our 2D data set. The green curve shows our method with temporal coherence loss, while the pink curve generated without it. The same data is shown in frequency space in (b). where the red curve represents the frequency the data with temporal loss, and the blue curve the frequency of the data without. This graph highlights the reduced amount of high frequency changes in the latent space with temporal loss, esp. in frequency space, where the red curve almost entirely lies helow the blue onc. (c) contains frequency information for the latent space content of the same 100 patch sequences, but in random order. In this case, the blue and red curve both contain significant amounts of high-frequencies. L..,, our method reliably identifies strongly changing inputs.
of the target points, that is also visible in terms of an increased Ls loss in Table While Fig. 5p) significantly reduces motions, and leads to an improved shape as well as Ls loss, its motions are overly constrained. E.g., at the bottom of the shown patch, the generated points should follow the black input points, but in (b) the generated points stay in place. In addition, the lack of permutation invariance leads to an undesirable clustering of generated points in the patch center. Both problems are removed with LEV in Fig. 5b), which still contains small scale jittering motions, unfortunately. These are removed by LEΛ in Fig. ()), which shows the result of our full algorithm. The success of our approach for dynamic output sizes is also shown in the L column of Table [ which contains an Le error w.r.t. ground truth size of the outputs.
Temporally Coherent Features A central goal of our work is to enable the learning of features that remain stable over time. To shed light on how our approach influences the established latent space, we analyze its content for different inputs. The latent space in our case consists of a 256-dimensiona. vector that contains the features extracted by the first set of layers of our network. Fig. 7 contains a qualitative example for 100 randomly selected patch sequences from our test data set, where we collect input data by following the trajectory of each patch center for 50 time steps extract coherent data sets. Fig. (7)) shows the averaged latent space content over time for these sequences. While the model trained with temporal coherence (green curve) is also visually smoother, the difference becomes clearer when considering temporal frequencies. We measure averaged frequencies of the latent space dimensions over time, as shown in Fig. (p.c). We quantify the differences by calculating the integral of the frequency spectrum f. weighted by the frequency x to emphasize high frequencies. i.e S x- ((rdd. Hence, small values are preferred. As shown in Fig. 7p), the version trained without our loss formulations contains significantly more high frequency content. This is also reflected in the weighted integrals, which are 36.56 for the method without temporal loss, and 16.98 for the method with temporal loss. To verify that our temporal model actually establishes stable temporal latent space instead of ignoring temporal information altogether, we evaluate the temporal frequencies for the same 100 inputs above, but with a randomized order over time. In this case, our model correctly identifies the incoherent inputs, and yields similarly high frequencies as the regular model with 28.44 and 35. 35.24, respectively. More details in Appendix C
7

---

Published as a conference paper at ICLR 2020
Figure 8: Evaluation of the temporal stability for generated point clouds, in red with our temporal loss formulation, blue without. Graph (a) shows the temporal change of the point density (Ist derivative), while (b) shows the 2nd derivative. In (c) and (d) the error of the 1st and 2nd derivatives of the positions w.r.t. ground ground-truth reference points is shown.
In addition, we evaluated the changes of generated outputs over time w.r.t. ground truth motion. For this we mapped the generated point clouds Y = [ 822 .... V% for 100 frames to evenly and dense sampled ground-truth points the original mesh Y {yy 3/2.... ..... y J (the moving man shown in Fig. D. This gives us a dense correlation between the data and the generated point clouds. For the mapping we used an assignment based on nearest neighbors: y + y. Using y we divide ý into n subsets Y, = {ūا r(ū;) = yi which correlate with the corresponding ground-truth points. For each subset we can now compute the mean position [ Eger: D and the sample density |Y:| measured by the number of points assigned to a ground-truth sample position. The temporal change of these values of particular interest. The change of the mean positions should correspond to the ground-truth changes, while the change of the density should be one. We have evaluated the error of the first and second derivative of positions, Table 2}. As can be seen from the plots, our method leads to clear improvements for all measured quantities. The individual spikes that are visible for both versions in the position errors (c,d) most likely correspond to sudden changes of the input motions for which our networks undershoots by producing a smooth version of the motion.
well the first and second derivative of density (see Fig. 3 and
w/o
with
w/o
with
Velocity 0.043
0.024
Variance of 1st Derivative
0.016
0.00013
Acceleration
0.078
0.043
Variance of 2nd Derivative
0.038
0.00017
Table 2:
Measurements averaged over 100 frames for version of our network without temporal loss ("w/o") and with our full temporal loss formulation ("with"). The left table shows the results for the error evaluation of the velocity and acceleration, whereas in the right table one can see the variance the density derivatives.
3D Results Our patch- baschb approach currently relies on a decomposition of the input volumes into patches over time, as outlined in Appendix [ As all of the following results involve temporal data, full sequences are provided the accompanying video. We apply our method to several complex 3D models to illustrate its performance. Fig. Iisows the input as well as several frames generated with our method for an animation of a spider. Our method produces an even and temporally stable reconstruction of the object. In comparison, Fig. (P)) shows the output from the previous work architecture (Yu et al. 2018a). It exhibits uneven point distributions and outliers, e.g., above the legs of the spider, in addition to uneven motions.
A second example for a moving human figure is shown in Fig. In both examples, our network covers the target shape much more evenly despite using much fewer points, as shown in Table Thanks to the flexible output size of our network, it can adapt to sparsely covered regions by generating correspondingly fewer outputs. The previous work architecture, with its fixed output size, needs to concentrate the fixed number of output points within the target shape, leading to an unnecessarily large point count. In order to demonstrate the flexibility of our method, we also apply it to a volumetric moving point cloud obtained from a liquid simulation. Thanks to the patch-based evaluation of our network, is agnostic to the overall size of the input volume. In this way, it can be used to generate coherent sets with millions of points. These examples also highlight our method's capabilities for generalization. While the 3D model was only trained on data from physics simulations, as outlined above, it learns stable features that can be flexibly applied to volumetric well as to surface-basee data. The metrics in Table show that for both 2D and 3D cases, our method leads to significantly improved quality, visible in lower loss values for spatial well as temporal terms.
8

---

Published as a conference paper at ICLR 2020
bj
Figure 9: Our method applicd to an animation a moving spider. (a) Input point cloud, (b) three frames of our method, (c) detail from previous work (top) and our method (bottom). Note that our method at the bottom preserves the shape with fewer outliers, and leads to more even distribution of points, despite generating fewer points in total (see Table
Input points
P.W. output points
P.W.,, factor
Ours, output points
Ours, factor
Spider
7,900
3,063,704
387.81
251,146
31.79
Moving person Liquid
10,243
5,224,536
510.06
367,385
35.87
513,247
6,430,984
12.53
Table 3: Point counts for the 3D examples of our video. Input counts together with output counts for previous work (P. and our proposed network shown. Factor columns contain increase in point size from in- output. As previous work cannot handle flexible output counts, fixed number of points generated per patch, leading to a huge number of redundant points. However, our network flexibly adapts the output size and leads to significantly smaller number of generated points that cover the object or volume more evenly.
Another interesting field of application for our algorithm are physical simulations. Complex simula- tions such as fluids, often employ particle-brr representations. On the one hand, the volume data much larger than surface-basee data, which additionally motivates our dynamic output. On the other hand, time stability plays very important role for physical phenomena. Our method produces detailed outputs for liquids, as be seen in supplemental video.
Convergence graphs for the different versions are shown in Fig. 12 the supplemental material. These graphs show that method not only successfully leads to very low errors in terms of temporal coherence, but also improves spatial accuracy. The final values of Ls for the 2D case are below 0.05 for our algorithm, compared to almost 0.08 for previous work. For 3D, our approach yields 0.04 average, in contrast to ca. 0.1 for previous work.
CONCLUSION
We have proposed first method infer temporally coherent features for point clouds. This is made possible by novel loss function for temporal coherence in combination with enabling flexible truncation of the results. addition we have shown that crucial prevent static patterns as easy-to-reach local minima for the network, which we avoid with the proposed mingling loss term. Our super-resolution results above demonstrate that our approach takes an important first step towards flexible deep learning methods for dynamic point clouds.
Looking ahead, our method could also be flexibly combined with other network architectures or could be adopted for other applications. Specifically, combination with PSGN (Fan et al. 2016) could be used to generate point clouds from image sequences instead single images. Other conceivable applications could employ methods like Dahnert al. (2019) with our approach for generating animated meshes. Due to the growing popularity and ubiquity of scanning devices will, e.g., be interesting to investigate classification tasks 3D scans over time future work. Apart from that, physical phenomena such as elastic bodies and fluids (Li et al. (019)) can likewise be represented in Lagrangian manner, and pose interesting challenges and complex spatio-temporal changes.
ACKNOWLEDGMENTS
This work is supported by grant TH 2034/1-1 of the Deutsche Forschungsgemmiinschat (DFG).

---

Published as a conference paper at ICLR 2020
REFERENCES
R Qi Charles, Hao Su, Mo Kaichun, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. pages 77-85, 07 2017. doi: 10. 109//CVRR.201..11.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning point sets in metric space. In Advances in Neural Information Processing Systems, pages 5099-5108. Curran Associates, Inc., 2017.
Pedro Hermosilla, Tobias Ritschel, Pere-Pau Vázquez, Alvar Vinacua, and Timo Ropinski. Monte carlo convolution for learning non-uniformly sampled point clouds. ACM Trans. Graph., 37(6): 233:1-235:12 12, December 2018. doi: 10. 10.11455/222227.225511..
Binh-Son Hua, Minh-Khoi Tran, and Sai-Kit Yeung. Pointwise convolutional neural networks. In Computer Vision and Pattern Recognition (CVPR), 2018.
Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. Pu-net: Point cloud upsampling network. In CVPR, 2018a. 8a.
Matan Atzmon, Haggai Maron, and Yaron Lipman. Point convolutional neural networks by extension operators. ACM Trans. Graph. 37(4):71:1-71:12 July 2018. doi: 10.1 10.1145/197511..221.33
Lequan Yu, Xianzhi Li, Chi-Wing Fu. Daniel Cohen-Or. and Pheng-Amn Heng. Ec-net: an edge-aware point set consolidation network. In European Conference on Computer Vision, pages 398-44 414. Springer, 2018b.
Siamak Ravanbakhsh, Jeff G. Schneider, and Barnabás Póczos. Deep learning with sets and point clouds. CoRR, abs/1611 L.04500, 2016.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan R. Salakhutdinov, and Alexander J. Smola. Deep sets. In Advances in Neural Information Processing Systems, pages 3394-3404, 2017.
Roman Klokov and Victor S. Lempitsky. Escape from cells: Deep kd-networks for the recognition of 3d point cloud models. In IEEE International Conference on Computer Vision, ICCV 2017, Venice Italy, October 22-29, 2017, pages 863-872, 2017. doi: 10. 10.1109/CCC.2011..999
Riccardo Roveri, A. Cengiz Öztireli, Ioana Pandele, and Markus Gross, Pointpronets: Consolidation of point clouds with convolutional neural networks. Computer Graphics Forum, 37(2):87-99, 2018. doi: (0.111 13344.
Paul Guerrero, Yanir Kleiman, Maks Ovsjanikov, and Niloy J. Mitra. PCPNet: Learning local shape properties from raw point clouds. Computer Graphies Forum, 37(2):75-85, 2018. doi:
111111/cgf.13333
Kangxue Yin, Hui Huang, Daniel Cohen-Or, and Hao Zhang. P2p-net: Bidirectional point dis- placement net for shape transform. ACM Trans. Graph., 37(4):152:1 July 2018. doi: 10.1 00.1145//197517.2222255
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, and Jan Kautz. Splatnet: Sparse lattice networks for point cloud processing. In CVPR, pages 2530-2539, IEEE Computer Society, 2018.
Weiyue Wang, Ronald Yu, Qiangui Huang, and Ulrich Neumann. Sgpn: Similarity group proposal network for 3d point cloud instance segmentation. pages 2569-2578, 06 2018a. doi: 10.1 109/ CVPR.2011 2018.00272.
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spidercnn: Deep learning on point sets with parameterized convolutional filters. In ECCV (8), volume 11212 of Lecture Notes in Computer Science, pages 90-105. Springer, 2018.
Wenxuan Wu, Zhongang Qi, and Fuxin Li. Pointcony: Deep convolutional networks on 3d point clouds. CoRR, 1es//1111.0726, 2018.
10

---

Published as a conference paper at ICLR 2020
Jiaxin Li, Ben M. Chen, and Gim Hee Lee. So-net: Self Self-organizing network for point cloud analysis. In CVPR, 2018a.
Xiaoqing Ye, Jiamao Li, Hexiao Huang, Liang Du, and Xiaolin Zhang. 3d recurrent neural networks with context fusion for point cloud semantic segmentation. In ECCV, 2018.
Groh, P. Wieschollek, and H. P. A. Lensch. Flex-convolution (million-scale point-cloud learning beyond grrd-wo In Computer Vision ACCV 2018 14th Asian Conference Computer Vision, December 2018.
Loic Landrieu and Martin Simonovsky, Large-scale point cloud semantic segmentation with super- point graphs. In CVPR, 2018
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, and Federico Tombari. Fully- convolutional point networks for large-scale point clouds. In Computer Vision ECCV 2018, pages 625-640. Springer International Publishing, 2018.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph enn for learning on point clouds. CoRR, abs/1801 .07829, 2018b.
Huan Lei, Naveed Akhtar, and Ajmal Mian. Spherical convolutional neural network for 3d point clouds. CoRR, htss1805500872. 2018.
Yingxue Zhang and Michael G. Rabbat. A graph-cnn for 3d point cloud classification. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6279-6288 2018.
Mark B. Skouson. Ursa: neural network for unordered point clouds using constellations. CoRR, abs/ abs/1808..0m 04844, 2018.
Martin Simon, Stefan Milz, Karl Amende, and Horst-Michael Groß. Complex-yolo: Real-time 3d object detection on point clouds. CoRR, abs//883.00613 2018.
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR, 2018
Liuhao Ge, Yujun Cai, Junwu Weng, and Junsong Yuan. Hand pointnet: 3d hand pose estimation using point sets. In CVPR, 2018.
Nathaniel Thomas, Tess Smidt, Steven M. Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-equiiarraat neural networks for 3d point clouds. CoRR, abs/1802.082111 2018.
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In CVPR, 2018.
Haowen Deng, Tolga Birdal, and Slobodan Ilic. Ppfnet: Global context aware local features for robust 3d point matching. In CVPR, pages 195-205. IEEE Computer Society, 2018.
Yongbin Sun, Yue Wang. Ziwei Liu, Joshua E. Siegel, and Sanjay E. Sarma. Pointgrow: Autoregres- sively learned point cloud generation with self-attention. CoRR, abs/1810.00591, 2018.
Chun-Liang Li, Manzil Zaheer, Yonghui Zhang, Barnabás Póczos, and Ruslan Salakhutdinov. Point cloud gan. CORR, abs/1810.05795, 2018b.
Xingyu Liu, Charles Ruizhongtai Qi, and Leonidas Guibas. Learning scene flow in 3d point clouds. CoRR, abs/1806.01411 2018.
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas J. Guibas. Representation learning and adversarial generation of 3d point clouds. CoRR, Abs/1707.02399 2017.
Haoqiang Fan, Hao Su, and Leonidas J. Guibas. A point set generation network for 3d object reconstruction from a single image. CoRR, abs/ Abss/611.00003, 2016. URL hattp://axxiv.oom abs/ 661 60603
11

---

Published as a conference paper at ICLR 2020
Manuel Dahnert, Angela Dai, Leonidas Guibas, and Matthias Nießner. Joint embedding of 3d scan and cad objects. In ICCV 2019, 2019.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICLR, 2019.
Markus Ihmsen, Jens Cornelis, Barbara Solenthaler, Christopher Horvath, and Matthias Teschner. Implicit incompressible sph. IEEE Transactions on Visualization and Computer Graphics, 20(3): 426-435, 2014.
12

---

Published as a conference paper at ICLR 2020
Tranquil Clouds: Neural Networks for Learning Temporally Co- herent Features in Point Clouds, Supplemental Material
A TRAINING AND EVALUATION MODALITIES
Data Generation We employ a physical simulation to generate our input and output pairs for training. This has the advantage that it leads to a large variety of complex motions, and gives full control of the generation process. More specifically, we employ the IISPH (Ihmsen et al.] 2014) algorithm, a form of Lagrangian fluid simulator that efficiently generates incompressible liquid volumes. These simulations also have the advantage that they inherently control the density of the point sampling thanks to their volume conserving properties. In order to generate input pairs for training. we randomly sample regions near the surface and extract points with a given radius around a central point. This represents the high-resolution target. To compute the low- low-resolution input, we downsample the points with Poisson-disk sampling to compute a point set with the desired larger spacing. In order to prevent aliasing from features below the coarse resolution, we perform a pass of surface fairing and smoothing before downsampling. Due to the large number of patches that can be extracted from these simulations, we did not find it necessary to additionally augment the generated data sets. Examples of the low- and high-resolution pairs are shown in the supplemental material.
Below we will demonstrate that models trained with this data can be flexibly applied to moving surface data as well as new liquid configurations. The surface data is generated from animated triangle meshes that resampled with bieubic interpolation in order match a chosen average per-point area. This pattern was generated once and then propagated over time with the animation. When applying our method new liquid simulations, we do not perform any downsampling, but rather use all points of a low-resolution simulation directly, as a volumetric re-sampling over time is typically error prone, and gives incoherent low resolution inputs.
Given a moving point cloud, we decompose it into temporally coherent patches in the following manner: We start by sampling points via Poisson-disk sampling in a narrow band around the surface, e.g., based on signed distance function computed from the input cloud. These points will persist as patch centers over time, unless they move too close to others, or too far away from the surface, which triggers their deletion. In addition, we perform several iterations for every new frame to sample new patches for points in the input cloud that are outside all existing patches. Note that this resampling of patches over time happens instantaneously in our implementation. While a temporal fading could easily be added, we have opted for employing transitions without fading, in order to show as much of the patch content possible.
Network Architecture and Training Our architecture heavily relies on a form of hicrarchical point-based convolutions. Le.. the network extracts features for subset of the points and their nearest neighbors. For the point convolution, we first select a given number of group centers that are evenly distributed points from given imput cloud. For each group center, we then search for certain number of points within a chosen radius (a fraction of the [-1,1] range). This motivates our choice for a coordinate far outside the regular range for the padded points from Sec. 3.2 They are too far away from all groups by construction, so they are filtered out without any additional overhead. In this way, both feature extraction and grouping operations work flexibly with the varying input sizes. Each group is then processed by a PointNet-like sub-structure (Qi Charles et al. 2017). yielding one feature vector per group.
The result is set of feature vectors and the associated group position, which can be interpreted as a new point cloud to repeatedly apply a point convolution. In this way, the network extracts increasingly abstract and global features. The last set of features is then interpolated back to the original points of the input. Afterwards a sub-pixel convolution layer is used to scale up the point cloud extended with features and finally the final position vectors are generated with the help of two additional shared, fully-connected layers. While we keep the core network architecture unmodified to allow for comparisons with previous work, important distinction of our approach is the input and output masking, as described in Sec. 3.2
Our point data was generated with mean point spacing, i.e., Poisson disk radius, of 0.5 units. For the 2D tests, an upscaling factor of r = 9 was used. For this purpose, patches with a diameter of were extracted from the low-resolution data and patches with diameter of 15 from the high-resolution
13

---

Published as a conference paper at ICLR 2020
data. We used the thresholds Xmaz 100 and Pena 900. For the loss, we used y 10, μ 10, and V/ = 0.001. The network was trained with 5 epochs for a data set with 185k pairs, and a batch size of 16, the learning rate 0.001 with decay of 0.003. For the 3D results below, the scaling factor r was set to 8. The diameter of the patches was 6 for the low-resolution data and 12 for the high-resolution data, with kmaz 1280 and Amaz 10240. The loss parameters were Y = / 5, with // = 0.001. Learning rate and decay were the same for training, but instead we used 10 epochs with 54k patches in 3D, and a batch size of 4.
B NETWORK ARCHITECTURE DETAILS
The input feature vector is processed in the first part of our network, which consists of four point convolutions. We use (ng, [1......u Ia)) to represent level with Ng groups of radius rg and [[.....44 ta] the d cully-connected layers with the width (i = 1..., d). The parameters we use are (kmar, 0.25, [32, 32, 64)). (Kmar/2 22, 0.5, [64, 64, 128)), 16mee/4,000 [128, 128, 2561) and (kmaar/8, 0.7, [256, (256, 256, 512)). We then use interpolation layers to distribute the features of each convolution level among the input points. In this step, we reduce the output of each convolution layer with one shared, fully-comma connectd layer per level, to a size of 64 and then distribute the features to all points of the input point cloud depending on their position. This extends the points of our original point cloud with 256 features. Fig. 11 shows a visual overview of the data flow in our network.
Afterwards, we process the data in separate branches consisting of two shared, fully interconnected layers with 256 and 128 nodes. The output is then processed with two shared fully-connecten layers of 64 and 3 nodes. Finally, we add resulting data to the input positions that have been repeated times. This provides an additional skip connection which leads to slightly more stable results. All convolution layers and fully interconnected layers use tanh() activation function.
For the input feature vector, we make use of additional data fields in conjunction with the point positions. Our network also accepts additional features such as velocity, density and pressure of the SPH simulations used for data generation. For inputs from other sources, those values could be easily computed with suitable SPH interpolation kernels. In practice, we use position, velocity and pressure fields. Whereas the first two are important (as mentioned in Sec. 3.I, the pressure fields turned out to have negligible influence.
C FREQUENCY EVALUATION OF LATENT SPACE
In this section we give details for the frequency evaluation of Sec. In order to measure the stability of the latent space against temporal changes, we evaluated the latent space of our network with and without temporal loss, once for 100 ordered patch sequences and once for 100 un-ordered ones. The central latent space of our network consists of the features generated by the point-convolution layers in the first part of the network and is 256 dimensional (see Fig. 11 To obtain information about its general behavior,
average the latent space components over all 100 patch sequences, subtract the mean, and normalize the resulting vector w.r.t. maximum value for each data set. The result is a time sequence of scalar values representing the mean deviations of the latent space. The Fourier transform of these vectors f, are shown in Fig. 7 and were used to compute the weighted frequency content [ x f(xr)dr. Here, large values indicate strong temporal changes of the latent space dimensions. The resulting values are given in the main document, and highlight the stability of the latent space learned by our method.
D TRAINING DATA AND GRAPHS
Two examples with ground truth points and down-sampled input versions are shown in Fig. 10
Additionally, Fig. 12 shows loss graphs for the different versions shown in the main text: 2D previous work, our full algorithm in 2D, as well both cases for 3D. The mingling loss L. is only shown as reference for the previous work versions, but indicates the strong halo-like patterns forming for the architectures based on previous work.
14

---

Published as a conference paper at ICLR 2020
Figure 10: Examples from our synthetic data generation process. In both sections (a) and (b) high resolution reference frame is shown in purple, and in green the down-sampled low resolution frames generated from it. The training data is generated by sampled patches from these volumes.
15

---

Published as a conference paper at ICLR 2020
Eaazz 64
Reaas 128)
Kanex 235
Noint.Net
Not
PointtNet
Foint.ett
Pmus 512) 16
Smaze points
(pos, vel, dons, press)
MLP
MLP
MLP
MLP
Fimax
64
Rioox 64) 16
interpollate
interpoolate
sstorppoote
sistorpolte
6d
branches
imppil
256 Jeutures
728
728
Pinoz Girne
728
oulpal
imipul position
Figure 11: An overview of network architecture. The first row shows the hierarchical point convolutions, while the bottom illustrate the processing of extracted features until the final output point coordinates generated.
Yay Pa
Oue
2.
Henden
Oanx
Figure Convergence plots the training runs our different and versions. The combined only illustrates convergence behavior for each method separately, as weights and terms differ across the four variants. Las for previous work is not minimized, and only given for reference.
16


Reviews:
[{"review_id": "SJlE85gmcS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the task of learning temporally stable features for point clouds with an application to upsampling point clouds. Learning point-based descriptors has been a major topic of research in the recent vision and graphics meetings, where approaches have been proposed focusing semantic labeling, geometry-oriented tasks (e.g. normal estimation), and point-based graphics. However, as the paper states, and to the best of my knowledge, no methods have been proposed to learn features in fourth dimension in a temporally stable way. Thus, the very topic of research is significantly novel and promising. \n\nThe authors consider a combination of loss functions and train a neural point-based network to learn the features. To stabilize training, the authors carefully study the effect of a series of loss functions, including well-known EMD, losses ensuring slow changes in positions, velocities, and accelerations, as well as a mingling loss to ensure a more uniform spatial point distribution on the output shape. The studied losses are very logical to implement as one aims to ensure that the output should satisfy a temporally smooth motion pattern. \n\nThe experimental results provide a clear view of the proposed approach and demonstrate that combining the studied objectives function with a known point-based learning approach leads to a temporally stable feature representation per-point. An ablation study further helps to validate the proposed approaches step-by-step.\n\nTo sum up, I believe paper should clearly be accepted, as (1) the work addresses a novel point-based learning task, (2) the research methodology is convincingly presented, and (3) the results provide a clear demonstration of the feasibility of the proposed task.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1emRfD0tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposed a deep network for point cloud sequence super-resolution/upsampling. Building on the basis of pointNet and PU-net, the main contribution of the paper is identifying the problem of temporal incoherence in the process of upsampling a point cloud shape representation as well as a training loss to encourage temporal coherence. In the cases showed in the paper, the proposed method seems effective comparing to previous work which is not done on sequence data. My main concern about the work is that the experimental evaluation is limited.\n\nStrengths:\nInteresting problem and novel idea.\nThe proposed method is technically sound. From the provided results, the newly introduced training loss seems effective: the result sequences are visually more plausible and smooth.\nWeaknesses:\nQualitative results are limited and in most cases seemingly simple. In the paper as well as the companion video, there are very few examples provided. The scale of the evaluation demonstration is not convincing enough for the readers that this work could be generalized to more complicated testing scenarios.\nQuantitative results are also limited. Since the method is handling the coherence of shape deformation over time, it would be much more convincing and helpful to introduce a dense-correspondence evaluation as a benchmark. For example, one can create ground-truth correspondence from parametric morphable models and evaluate the coherence of the sequence by comparing the generated results with the ground truth.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkeSwCWwqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tries to learn temporally stable representations for point-based data sets and focus on varying size and dynamic point sets, and demonstrate its usefulness in the context of super-resolution. To deal with a difficult target that dynamically moves and deforms over time with variable input and output size, they take a novel temporal loss function for temporally coherent point set generation and siamese network setup for temporal loss calculation. Their novel temporal loss is based on EMD to minimize differences between an estimated point cloud and a desired super-resolution point cloud. The discussion and evolution on multiple loss functions are mostly well done. Except spatial loss is considered, taking the ground truth acceleration and estimated velocity into account is beneficial to this task. Their main contribution is taking permutation invariant loss terms and a siamese training setup and generator architecture, enabling improved output variance by allowing for dynamic adjustments of the output size, and identifying a specialized form of mode collapse for temporal point networks. \nThey perform an empirical study of their temporal loss function on the generated data set and apply the proposed method to some complex 3D models to conclude the superior performance of temporal loss formulation in contrast to previous work.\n \nOverall, this paper has some significant points on point cloud super-resolution, with the caveat for some clarifications on the theory and experiments. Given these clarifications in an author response, I would be willing to increase the score.\n \nWhen the input moves slowly enough, the point cloud can be considered static. Can the proposed temporal loss outperform other works under this condition? \n \nOnly one previous work PU-Net based on PointNet++ is compared in the paper, I would like to see more discussion on applying the proposed temporal loss with other point-based algorithms.\n \nI am very curious about the effect on different choices of weighting terms hyperparameters in temporal loss and predefined upsampling factor r.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides an improved method for deep learning on point clouds.  Reviewers are unanimous that this paper is acceptable, and the AC concurs. 
Decision: Accept (Spotlight)