Tranquil Clouds: Neural Networks for Learning Temporally Coherent Features in Point Clouds

Point clouds , as a form of Lagrangian representation , allow for powerful and flexible applications in a large number of computational disciplines . We propose a novel deep-learning method to learn stable and temporally coherent feature spaces for points clouds that change over time . We identify a set of inherent problems with these approaches : without knowledge of the time dimension , the inferred solutions can exhibit strong flickering , and easy solutions to suppress this flickering can result in undesirable local minima that manifest themselves as halo structures . We propose a novel temporal loss function that takes into account higher time derivatives of the point positions , and encourages mingling , i.e. , to prevent the aforementioned halos . We combine these techniques in a super-resolution method with a truncation approach to flexibly adapt the size of the generated positions . We show that our method works for large , deforming point sets from different sources to demonstrate the flexibility of our approach . 1 INTRODUCTION . Deep learning methods have proven themselves as powerful computational tools in many disciplines , and within it a topic of strongly growing interest is deep learning for point-based data sets . These Lagrangian representations are challenging for learning methods due to their unordered nature , but are highly useful in a variety of settings from geometry processing and 3D scanning to physical simulations , and since the seminal work of Qi Charles et al . ( 2017 ) , a range of powerful inference tasks can be achieved based on point sets . Despite their success , interestingly , no works so far have taken into account time . Our world , and the objects within it , naturally move and change over time , and as such it is crucial for flexible point-based inference to take the time dimension into account . In this context , we propose a method to learn temporally stable representations for point-based data sets , and demonstrate its usefulness in the context of super-resolution . An inherent difficulty of point-based data is their lack of ordering , which makes operations such as convolutions , which are easy to perform for Eulerian data , unexpectedly difficult . Several powerful approaches for point-based convolutions have been proposed ( Qi et al. , 2017 ; Hermosilla et al. , 2018 ; Hua et al. , 2018 ) , and we leverage similar neural network architectures in conjunction with the permutation-invariant Earth Mover ’ s Distance ( EMD ) to propose a first formulation of a loss for temporal coherence . In addition , several works have recognized the importance of training point networks for localized patches , in order to avoid having the network to rely on a full view of the whole data-set for tasks that are inherently local , such as normal estimation ( Qi Charles et al. , 2017 ) , and super-resolution ( Yu et al. , 2018a ) . This also makes it possible to flexibly process inputs of any size without being limited by memory requirements . Later on we will demonstrate the importance of such a patch-based approach with sets of changing cardinality in our setting . A general challenge here is to deal with varying input sizes , and for super-resolution tasks , also varying output sizes . Thus , in summary we target an extremely challenging learning problem : we are facing permutation-invariant inputs and targets of varying size , that dynamically move and deform over time . In order to enable deep learning approaches in this context , we make the following key contributions : Permutation invariant loss terms for temporally coherent point set generation ; A Siamese training setup and generator architecture for point-based super-resolution with neural networks ; Enabling improved output variance by allowing for dynamic adjustments of the output size ; The identification of a specialized form of mode collapse for temporal point networks , together with a loss term to remove them . We demonstrate that these contributions together make it possible to infer stable solutions for dynamically moving point clouds with millions of points . More formally , we show that our learning approach can be used for generating a point set with an increased resolution from a given set of input points . The generated points should provide an improved discretization of the underlying ground truth shape represented by the initial set of points . For the increase , we will target a factor of two to three per spatial dimension . Thus , the network has the task to estimate the underlying shape , and to generate suit- able sampling positions as output . This is generally difficult due to the lack of connectivity and ordering , and in our case , positions that move over time in combination with a changing number of input points . Hence it is crucial that the network is able to establish a temporally stable latent space representation . Although we assume that we know correspondences over time , i.e. , we know which point at time t moved to a new location at time t+ ∆t , the points can arbitrarily change their relative position and density over the course of their movement , leading to a substantially more difficult inference problem than for the static case . 2 RELATED WORK . Deep learning with static point sets was first targeted in PointNet ( Qi Charles et al. , 2017 ) via order-invariant networks , while PointNet++ ( Qi et al. , 2017 ) extended this concept to generate features for localized groups similar to a convolution operation for grid-based data . This concept can be hierarchically applied to the generated groups , in order to extract increasingly abstract and global features . Afterwards , the extracted features can be interpolated back to the original point cloud . The goal to define point convolutions has been explored and extended in several works . The MCNN approach ( Hermosilla et al. , 2018 ) phrased convolution in terms of a Monte Carlo integration . PointCNN ( Hua et al. , 2018 ) defined a pointwise convolution operator using nearest neighbors , while extension-restriction operators for mapping between a point cloud function and a volumetric function were used in Atzmon et al . ( 2018 ) . The PU-Net ( Yu et al. , 2018a ) proposed a network for upsampling point clouds , and proposed a similar hierarchical network structure of PointNets along the lines of PointNet++ to define convolutions . Being closer to our goals , we employ this approach for convolutional operations in our networks below . We do not employ the edge-aware variant of the PU-Net ( Yu et al. , 2018b ) here , to keep it as simple and general as possible as we focus on temporal changes in our work . Permutation invariance is a central topic for point data , and was likewise targeted in other works ( Ravanbakhsh et al. , 2016 ; Zaheer et al. , 2017 ) . The Deep Kd-network ( Klokov and Lempitsky , 2017 ) defined a hierarchical convolution on point clouds via kd-trees . PointProNets ( Roveri et al. , 2018 ) employed deep learning to generate dense sets of points from sparse and noisy input points for 3D reconstruction applications . PCPNet ( Guerrero et al. , 2018 ) , as another multi-scale variant of PointNet , has demonstrated high accuracy for estimating local shape properties such as normal or curvature . P2PNet ( Yin et al. , 2018 ) used a bidirectional network and extends PointNet++ to learn a transformation between two point clouds with the same cardinality . Recently , the area of point-based learning has seen a huge rise in interest . One focus here are 3D segmentation problems , where numerous improvements were proposed , e.g. , by SPLATNet ( Su et al. , 2018 ) , SGPN ( Wang et al. , 2018a ) , SpiderCNN ( Xu et al. , 2018 ) , PointConv ( Wu et al. , 2018 ) , SONEt ( Li et al. , 2018a ) and 3DRNN ( Ye et al. , 2018 ) . Other networks such as Flex Convolution ( Groh et al. , 2018 ) , the SuperPoint Graph ( Landrieu and Simonovsky , 2018 ) , and the fully convolutional network ( Rethage et al. , 2018 ) focused on large scale segmentation . Additional areas of interest are shape classification ( Wang et al. , 2018b ; Lei et al. , 2018 ; Zhang and Rabbat , 2018 ; Skouson , 2018 ) and object detection ( Simon et al. , 2018 ; Zhou and Tuzel , 2018 ) , and hand pose tracking ( Ge et al. , 2018 ) . Other works have targeted rotation and translation invariant inference ( Thomas et al. , 2018 ) , and point cloud autoencoders ( Yang et al. , 2018 ; Deng et al. , 2018 ) . A few works have also targeted generative models based on points , e.g. , for point cloud generation ( Sun et al. , 2018 ) , and with adversarial approaches ( Li et al. , 2018b ) . It is worth noting here that despite the huge interest , the works above do not take into account temporally changing data , which is the focus of our work . A notable exception is an approach for scene flow ( Liu et al. , 2018 ) , in order to estimate 3D motion directly on the basis of point clouds . This work is largely orthogonal to ours , as it does not target generative point-based models . 3 METHODOLOGY . We assume an input point cloud X = { x1 , x2 , ... , xk } of size k ∈ [ 1 , kmax ] . It consists of points xi ∈ Rd , where d includes 3 spatial coordinates and optionally additional features . Our goal is to let the network fs ( X ) infer a function Ỹ which approximates a desired super-resolution output point cloud Y = { y1 , y2 , ... , yn } of size n ∈ [ 1 , nmax ] with yi ∈ R3 , i.e . fs ( X ) = Ỹ ≈ Y . For now we assume that the number of output points n is defined by multiplying k with a user-defined upsampling factor r , i.e . n = rk . Figure 2a ) illustrates the data flow in our super-resolution network schematically . We treat the upsampling problem as a local one , i.e. , we assume that the inference problem can be solved based on a spatially constrained neighborhood . This allows us to work with individual patches extracted from input point clouds . At the same time , it makes it possible to upsample adaptively , for example , by limiting the inference to relevant areas , such as complex surface structures . For the patch extraction we use a fixed spatial radius and normalize point coordinates within each patch to lie in the range of [ −1 , 1 ] . Our first building block is a measure for how well two point clouds represent the same object or scene by formulating an adequate spatial loss function . Following Achlioptas et al . ( 2017 ) , we base our spatial loss LS on the Earth Mover ’ s Distance ( EMD ) , which solves an assignment problem to obtain a differentiable bijective mapping φ : ỹ → y . With φ we can minimize differences in position for arbitrary orderings of the points clouds via : LS = min φ : ỹ→y ∑ ỹi∈Ỹ ‖ỹi − φ ( ỹi ) ‖22 ( 1 ) 3.1 TEMPORAL COHERENCE When not taking temporal coherence explicitly into account , the highly nonlinear and ill-posed nature of the super-resolution problem can cause strong variations in the output even for very subtle changes in the input . This results in significant temporal artifacts that manifest themselves as flickering . In order to stabilize the output while at the same time keeping the network structure as small and simple as possible , we propose the following training setup . Given a sequence of high resolution point clouds Y t , with t indicating time , we can compute a velocity V t = { vt1 , vt2 , ... , vtk } , where vti ∈ R3 . For this we use a finite difference ( yt+1i − yti ) , where we assume , without loss of generality , ∆t = 1 , i.e . the time step is normalized to one . For training , the low resolution inputs X can now be generated from Y via down-sampling by a factor of r , which yields a subset of points with velocities . Details of our data generation process will be given below . To train a temporally coherent network with the Y t sequences , we employ a Siamese setup shown in Figure 2b . We evaluate the network several times ( 3 times in practice ) with the same set of weights , and moving inputs , in order to enforce the output to behave consistently . In this way we avoid recurrent architectures that would have to process the high resolution outputs multiple times . In addition , we can compute temporal derivatives from the input points , and use them to control the behavior of the generated output . Under the assumption of slowly moving inputs , which theoretically could be ensured for training , a straightforward way to enforce temporal coherence would be to minimize the movement of the generated positions over consecutive time steps in terms of an L2 norm : L2V = n∑ i=1 ‖ỹt+1i − ỹ t i‖22 . ( 2 ) While this reduces flickering , it does not constrain the change of velocities , i.e. , the acceleration . This results in a high frequency jittering of the generated point positions . The jitter can be reduced by also including the previous state at time step t− 1 to constrain the acceleration in terms of its L2 norm : L2A = n∑ i=1 ‖ỹt+1i − 2ỹ t i + ỹ t−1 i ‖ 2 2 ( 3 ) However , a central problem of a direct temporal constraint via Equations ( 2 ) and ( 3 ) is that it consistently leads to a highly undesirable clustering of generated points around the center point . This is caused by the fact that the training procedure as described so far is unbalanced , as it only encourages minimizing changes . The network can not learn to reconstruct realistic , larger motions in this way , but rather can trivially minimize the loss by contracting all outputs to a single point . For this reason , we instead use the estimated velocity of the ground truth point cloud sequence with a forward difference in time , to provide the network with a reference . By using the EMD-based mapping φ established for the spatial loss in Equation ( 1 ) , we can formulate the temporal constraint in a permutation invariant manner as LEV = n∑ i=1 ‖ ( ỹt+1i − ỹ t i ) − ( φ ( ỹt+1i ) − φ ( ỹ t i ) ) ‖22 . ( 4 ) Intuitively , this means the generated outputs should mimic the motion of the closest ground truth points . As detailed for the L2-based approaches above , it makes sense to also take the ground truth acceleration into account to minimize rapid changes of velocity over time . We can likewise formulate this in a permutation invariant way w.r.t . ground truth points via : LEA = n∑ i=1 ‖ ( ỹt+1i − 2ỹ t i + ỹ t−1 i ) − ( φ ( ỹ t+1 i ) − 2φ ( ỹ t i ) + φ ( ỹ t−1 i ) ) ‖22 . ( 5 ) We found that a combination of LEV and LEA together with the spatial loss LS from Eq . 1 provides the best results , as we will demonstrate below . First , we will introduce the additional loss terms of our algorithm .

Summary: The paper tries to learn temporally stable representations for point-based data sets and focus on varying size and dynamic point sets, and demonstrate its usefulness in the context of super-resolution. To deal with a difficult target that dynamically moves and deforms over time with variable input and output size, they take a novel temporal loss function for temporally coherent point set generation and siamese network setup for temporal loss calculation. Their novel temporal loss is based on EMD to minimize differences between an estimated point cloud and a desired super-resolution point cloud. The discussion and evolution on multiple loss functions are mostly well done. Except spatial loss is considered, taking the ground truth acceleration and estimated velocity into account is beneficial to this task. Their main contribution is taking permutation invariant loss terms and a siamese training setup and generator architecture, enabling improved output variance by allowing for dynamic adjustments of the output size, and identifying a specialized form of mode collapse for temporal point networks.