Tranquil Clouds: Neural Networks for Learning Temporally Coherent Features in Point Clouds

Point clouds , as a form of Lagrangian representation , allow for powerful and flexible applications in a large number of computational disciplines . We propose a novel deep-learning method to learn stable and temporally coherent feature spaces for points clouds that change over time . We identify a set of inherent problems with these approaches : without knowledge of the time dimension , the inferred solutions can exhibit strong flickering , and easy solutions to suppress this flickering can result in undesirable local minima that manifest themselves as halo structures . We propose a novel temporal loss function that takes into account higher time derivatives of the point positions , and encourages mingling , i.e. , to prevent the aforementioned halos . We combine these techniques in a super-resolution method with a truncation approach to flexibly adapt the size of the generated positions . We show that our method works for large , deforming point sets from different sources to demonstrate the flexibility of our approach . 1 INTRODUCTION . Deep learning methods have proven themselves as powerful computational tools in many disciplines , and within it a topic of strongly growing interest is deep learning for point-based data sets . These Lagrangian representations are challenging for learning methods due to their unordered nature , but are highly useful in a variety of settings from geometry processing and 3D scanning to physical simulations , and since the seminal work of Qi Charles et al . ( 2017 ) , a range of powerful inference tasks can be achieved based on point sets . Despite their success , interestingly , no works so far have taken into account time . Our world , and the objects within it , naturally move and change over time , and as such it is crucial for flexible point-based inference to take the time dimension into account . In this context , we propose a method to learn temporally stable representations for point-based data sets , and demonstrate its usefulness in the context of super-resolution . An inherent difficulty of point-based data is their lack of ordering , which makes operations such as convolutions , which are easy to perform for Eulerian data , unexpectedly difficult . Several powerful approaches for point-based convolutions have been proposed ( Qi et al. , 2017 ; Hermosilla et al. , 2018 ; Hua et al. , 2018 ) , and we leverage similar neural network architectures in conjunction with the permutation-invariant Earth Mover ’ s Distance ( EMD ) to propose a first formulation of a loss for temporal coherence . In addition , several works have recognized the importance of training point networks for localized patches , in order to avoid having the network to rely on a full view of the whole data-set for tasks that are inherently local , such as normal estimation ( Qi Charles et al. , 2017 ) , and super-resolution ( Yu et al. , 2018a ) . This also makes it possible to flexibly process inputs of any size without being limited by memory requirements . Later on we will demonstrate the importance of such a patch-based approach with sets of changing cardinality in our setting . A general challenge here is to deal with varying input sizes , and for super-resolution tasks , also varying output sizes . Thus , in summary we target an extremely challenging learning problem : we are facing permutation-invariant inputs and targets of varying size , that dynamically move and deform over time . In order to enable deep learning approaches in this context , we make the following key contributions : Permutation invariant loss terms for temporally coherent point set generation ; A Siamese training setup and generator architecture for point-based super-resolution with neural networks ; Enabling improved output variance by allowing for dynamic adjustments of the output size ; The identification of a specialized form of mode collapse for temporal point networks , together with a loss term to remove them . We demonstrate that these contributions together make it possible to infer stable solutions for dynamically moving point clouds with millions of points . More formally , we show that our learning approach can be used for generating a point set with an increased resolution from a given set of input points . The generated points should provide an improved discretization of the underlying ground truth shape represented by the initial set of points . For the increase , we will target a factor of two to three per spatial dimension . Thus , the network has the task to estimate the underlying shape , and to generate suit- able sampling positions as output . This is generally difficult due to the lack of connectivity and ordering , and in our case , positions that move over time in combination with a changing number of input points . Hence it is crucial that the network is able to establish a temporally stable latent space representation . Although we assume that we know correspondences over time , i.e. , we know which point at time t moved to a new location at time t+ ∆t , the points can arbitrarily change their relative position and density over the course of their movement , leading to a substantially more difficult inference problem than for the static case . 2 RELATED WORK . Deep learning with static point sets was first targeted in PointNet ( Qi Charles et al. , 2017 ) via order-invariant networks , while PointNet++ ( Qi et al. , 2017 ) extended this concept to generate features for localized groups similar to a convolution operation for grid-based data . This concept can be hierarchically applied to the generated groups , in order to extract increasingly abstract and global features . Afterwards , the extracted features can be interpolated back to the original point cloud . The goal to define point convolutions has been explored and extended in several works . The MCNN approach ( Hermosilla et al. , 2018 ) phrased convolution in terms of a Monte Carlo integration . PointCNN ( Hua et al. , 2018 ) defined a pointwise convolution operator using nearest neighbors , while extension-restriction operators for mapping between a point cloud function and a volumetric function were used in Atzmon et al . ( 2018 ) . The PU-Net ( Yu et al. , 2018a ) proposed a network for upsampling point clouds , and proposed a similar hierarchical network structure of PointNets along the lines of PointNet++ to define convolutions . Being closer to our goals , we employ this approach for convolutional operations in our networks below . We do not employ the edge-aware variant of the PU-Net ( Yu et al. , 2018b ) here , to keep it as simple and general as possible as we focus on temporal changes in our work . Permutation invariance is a central topic for point data , and was likewise targeted in other works ( Ravanbakhsh et al. , 2016 ; Zaheer et al. , 2017 ) . The Deep Kd-network ( Klokov and Lempitsky , 2017 ) defined a hierarchical convolution on point clouds via kd-trees . PointProNets ( Roveri et al. , 2018 ) employed deep learning to generate dense sets of points from sparse and noisy input points for 3D reconstruction applications . PCPNet ( Guerrero et al. , 2018 ) , as another multi-scale variant of PointNet , has demonstrated high accuracy for estimating local shape properties such as normal or curvature . P2PNet ( Yin et al. , 2018 ) used a bidirectional network and extends PointNet++ to learn a transformation between two point clouds with the same cardinality . Recently , the area of point-based learning has seen a huge rise in interest . One focus here are 3D segmentation problems , where numerous improvements were proposed , e.g. , by SPLATNet ( Su et al. , 2018 ) , SGPN ( Wang et al. , 2018a ) , SpiderCNN ( Xu et al. , 2018 ) , PointConv ( Wu et al. , 2018 ) , SONEt ( Li et al. , 2018a ) and 3DRNN ( Ye et al. , 2018 ) . Other networks such as Flex Convolution ( Groh et al. , 2018 ) , the SuperPoint Graph ( Landrieu and Simonovsky , 2018 ) , and the fully convolutional network ( Rethage et al. , 2018 ) focused on large scale segmentation . Additional areas of interest are shape classification ( Wang et al. , 2018b ; Lei et al. , 2018 ; Zhang and Rabbat , 2018 ; Skouson , 2018 ) and object detection ( Simon et al. , 2018 ; Zhou and Tuzel , 2018 ) , and hand pose tracking ( Ge et al. , 2018 ) . Other works have targeted rotation and translation invariant inference ( Thomas et al. , 2018 ) , and point cloud autoencoders ( Yang et al. , 2018 ; Deng et al. , 2018 ) . A few works have also targeted generative models based on points , e.g. , for point cloud generation ( Sun et al. , 2018 ) , and with adversarial approaches ( Li et al. , 2018b ) . It is worth noting here that despite the huge interest , the works above do not take into account temporally changing data , which is the focus of our work . A notable exception is an approach for scene flow ( Liu et al. , 2018 ) , in order to estimate 3D motion directly on the basis of point clouds . This work is largely orthogonal to ours , as it does not target generative point-based models . 3 METHODOLOGY . We assume an input point cloud X = { x1 , x2 , ... , xk } of size k ∈ [ 1 , kmax ] . It consists of points xi ∈ Rd , where d includes 3 spatial coordinates and optionally additional features . Our goal is to let the network fs ( X ) infer a function Ỹ which approximates a desired super-resolution output point cloud Y = { y1 , y2 , ... , yn } of size n ∈ [ 1 , nmax ] with yi ∈ R3 , i.e . fs ( X ) = Ỹ ≈ Y . For now we assume that the number of output points n is defined by multiplying k with a user-defined upsampling factor r , i.e . n = rk . Figure 2a ) illustrates the data flow in our super-resolution network schematically . We treat the upsampling problem as a local one , i.e. , we assume that the inference problem can be solved based on a spatially constrained neighborhood . This allows us to work with individual patches extracted from input point clouds . At the same time , it makes it possible to upsample adaptively , for example , by limiting the inference to relevant areas , such as complex surface structures . For the patch extraction we use a fixed spatial radius and normalize point coordinates within each patch to lie in the range of [ −1 , 1 ] . Our first building block is a measure for how well two point clouds represent the same object or scene by formulating an adequate spatial loss function . Following Achlioptas et al . ( 2017 ) , we base our spatial loss LS on the Earth Mover ’ s Distance ( EMD ) , which solves an assignment problem to obtain a differentiable bijective mapping φ : ỹ → y . With φ we can minimize differences in position for arbitrary orderings of the points clouds via : LS = min φ : ỹ→y ∑ ỹi∈Ỹ ‖ỹi − φ ( ỹi ) ‖22 ( 1 ) 3.1 TEMPORAL COHERENCE When not taking temporal coherence explicitly into account , the highly nonlinear and ill-posed nature of the super-resolution problem can cause strong variations in the output even for very subtle changes in the input . This results in significant temporal artifacts that manifest themselves as flickering . In order to stabilize the output while at the same time keeping the network structure as small and simple as possible , we propose the following training setup . Given a sequence of high resolution point clouds Y t , with t indicating time , we can compute a velocity V t = { vt1 , vt2 , ... , vtk } , where vti ∈ R3 . For this we use a finite difference ( yt+1i − yti ) , where we assume , without loss of generality , ∆t = 1 , i.e . the time step is normalized to one . For training , the low resolution inputs X can now be generated from Y via down-sampling by a factor of r , which yields a subset of points with velocities . Details of our data generation process will be given below . To train a temporally coherent network with the Y t sequences , we employ a Siamese setup shown in Figure 2b . We evaluate the network several times ( 3 times in practice ) with the same set of weights , and moving inputs , in order to enforce the output to behave consistently . In this way we avoid recurrent architectures that would have to process the high resolution outputs multiple times . In addition , we can compute temporal derivatives from the input points , and use them to control the behavior of the generated output . Under the assumption of slowly moving inputs , which theoretically could be ensured for training , a straightforward way to enforce temporal coherence would be to minimize the movement of the generated positions over consecutive time steps in terms of an L2 norm : L2V = n∑ i=1 ‖ỹt+1i − ỹ t i‖22 . ( 2 ) While this reduces flickering , it does not constrain the change of velocities , i.e. , the acceleration . This results in a high frequency jittering of the generated point positions . The jitter can be reduced by also including the previous state at time step t− 1 to constrain the acceleration in terms of its L2 norm : L2A = n∑ i=1 ‖ỹt+1i − 2ỹ t i + ỹ t−1 i ‖ 2 2 ( 3 ) However , a central problem of a direct temporal constraint via Equations ( 2 ) and ( 3 ) is that it consistently leads to a highly undesirable clustering of generated points around the center point . This is caused by the fact that the training procedure as described so far is unbalanced , as it only encourages minimizing changes . The network can not learn to reconstruct realistic , larger motions in this way , but rather can trivially minimize the loss by contracting all outputs to a single point . For this reason , we instead use the estimated velocity of the ground truth point cloud sequence with a forward difference in time , to provide the network with a reference . By using the EMD-based mapping φ established for the spatial loss in Equation ( 1 ) , we can formulate the temporal constraint in a permutation invariant manner as LEV = n∑ i=1 ‖ ( ỹt+1i − ỹ t i ) − ( φ ( ỹt+1i ) − φ ( ỹ t i ) ) ‖22 . ( 4 ) Intuitively , this means the generated outputs should mimic the motion of the closest ground truth points . As detailed for the L2-based approaches above , it makes sense to also take the ground truth acceleration into account to minimize rapid changes of velocity over time . We can likewise formulate this in a permutation invariant way w.r.t . ground truth points via : LEA = n∑ i=1 ‖ ( ỹt+1i − 2ỹ t i + ỹ t−1 i ) − ( φ ( ỹ t+1 i ) − 2φ ( ỹ t i ) + φ ( ỹ t−1 i ) ) ‖22 . ( 5 ) We found that a combination of LEV and LEA together with the spatial loss LS from Eq . 1 provides the best results , as we will demonstrate below . First , we will introduce the additional loss terms of our algorithm .

Summary: The paper addresses the task of learning temporally stable features for point clouds with an application to upsampling point clouds. Learning point-based descriptors has been a major topic of research in the recent vision and graphics meetings, where approaches have been proposed focusing semantic labeling, geometry-oriented tasks (e.g. normal estimation), and point-based graphics. However, as the paper states, and to the best of my knowledge, no methods have been proposed to learn features in fourth dimension in a temporally stable way. Thus, the very topic of research is significantly novel and promising.