CloudLSTM: A Recurrent Neural Model for Spatiotemporal Point-cloud Stream Forecasting

1 INTRODUCTION . Point-cloud stream forecasting aims at predicting the future values and/or locations of data streams generated by a geospatial point-cloud S , given sequences of historical observations ( Shi & Yeung , 2018 ) . Example data sources include mobile network antennas that serve the traffic generated by ubiquitous mobile services at city scale ( Zhang et al. , 2019b ) , sensors that monitor the air quality of a target region ( Cheng et al. , 2018 ) , or moving crowds that produce individual trajectories . Unlike traditional spatiotemporal forecasting on grid-structural data , like precipitation nowcasting ( Shi et al. , 2015 ) or video frame prediction ( Wang et al. , 2018 ) , point-cloud stream forecasting needs to operate on geometrically scattered sets of points , which are irregular and unordered , and encapsulate complex spatial correlations . While vanilla Long Short-term Memories ( LSTMs ) have modest abilities to exploit spatial features ( Shi et al. , 2015 ) , convolution-based recurrent neural network ( RNN ) models , such as ConvLSTM ( Shi et al. , 2015 ) and PredRNN++ ( Wang et al. , 2018 ) , are limited to modeling grid-structural data , and are therefore inappropriate for handling scattered point-clouds . Leveraging the location information embedded in such irregular data sources , so as to learn important spatiotemporal features , is in fact challenging . Existing approaches that tackle the point-cloud stream forecasting problem can be categorized into two classes , both of which bear significant shortcomings : ( i ) methods that transform point-clouds into data structures amenable to processing with mature solutions , e.g. , grids as exemplified in Fig.1 ( Zhang et al. , 2019a ) ; and ( ii ) models that ignore the exact locations of each data source and inherent spatial correlations ( Liang et al. , 2018 ) . The transformations required by the former not only add data preprocessing overhead , but also introduce spatial displacements that distort relevant correlations among points ( Zhang et al. , 2019a ) . On the other hand , the latter are largely location-invariant , while recent literature suggests spatial correlations should be revisited over time , to suit series prediction tasks ( Shi et al. , 2017 ) . In essence , overlooking dynamic spatial correlations will lead to modest forecasting performance . Contributions . In this paper , we introduce Convolutional Point-cloud LSTMs ( CloudLSTMs ) , a new branch of recurrent neural network models tailored to geospatial point-cloud stream forecasting . The CloudLSTM builds upon a Dynamic Point-cloud Convolution ( DConv ) operator , which takes raw point-cloud streams ( both data time series and spatial coordinates ) as input , and performs dynamic convolution over these , to learn spatiotemporal features over time , irrespective of the topology and permutations of the point-cloud . This eliminates the need for the data preprocessing described above , and so avoids the distortions thereby introduced . The proposed CloudLSTM takes into account the locations of each data source and performs dynamic positioning at each time step , to conduct a deformable convolution operation over point-clouds ( Dai et al. , 2017 ) . This allows revising the spatiotemporal correlations and the configuration of the data points over time , and guarantees the location-invariant property is met at different steps . Importantly , the DConv operator is flexible , as it can be easily plugged into existing neural network models for different purposes , such as RNNs , LSTMs , sequence-to-sequence ( Seq2seq ) learning ( Sutskever et al. , 2014 ) , and attention mechanisms ( Luong et al. , 2015 ) . We evaluate our proposed architectures on four benchmark datasets , covering two spatiotemporal point-cloud stream forecasting applications : ( i ) antenna-level forecasting of data traffic generated by mobile services ( Zhang & Patras , 2018 ; Bega et al. , 2019 ) , leveraging metropolitan-scale mobile traffic measurements collected in two European cities for 38 popular mobile apps ; and ( ii ) forecasting six air quality indicators on two city clusters in China ( Zheng et al. , 2015 ) . These tasks represent important use cases of geospatial point-cloud stream forecasting . We combine our CloudLSTM with Seq2seq learning and an attention mechanism , then undertake a comprehensive evaluation on all datasets . The results demonstrate that our architecture can deliver precise long-term point-cloud stream forecasting in different settings , outperforming 12 different baseline neural network models in terms of four performance metrics , without any data preprocessing requirements . To the best of knowledge , the proposed CloudLSTM is the first dedicated neural architecture for spatiotemporal forecasting that operates directly on point-cloud streams . 2 RELATED WORK . Spatiotemporal Forecasting . Convolution-based RNN architectures have been widely employed for spatiotemporal forecasting , as they simultaneously capture spatial and temporal dynamics of the input . Shi et al. , incorporate convolution into LSTMs , building a ConvLSTM for precipitation nowcasting ( Shi et al. , 2015 ) . This approach exploits spatial information , which in turn leads to higher prediction accuracy . The ConvLSTM is improved by constructing a subnetwork to predict stateto-state connections , thereby guaranteeing location-variance and flexibility of the model ( Shi et al. , 2017 ) . PredRNN ( Wang et al. , 2017 ) and PredRNN++ ( Wang et al. , 2018 ) evolve the ConvLSTM by constructing spatiotemporal cells and adding gradient highway units . These improve long-term forecasting performance and mitigate the gradient vanishing problem in recurrent architectures . Although these solution work well for spatiotemporal forecasting , they can not be applied directly to point-cloud streams , as they require point-cloud-to-grid preprocessing ( Zhang et al. , 2019a ) . Feature Learning on Point-clouds . Deep neural networks for feature learning on point-cloud data are advancing rapidly . PointNet performs feature learning and maintains input permutation invariance ( Qi et al. , 2017a ) . PointNet++ upgrades this structure by hierarchically partitioning point-clouds and performing feature extraction on local regions ( Qi et al. , 2017b ) . VoxelNet employs voxel feature encoding to limit inter-point interactions within a voxel ( Zhou & Tuzel , 2018 ) . This effectively projects cloud-points onto sub-grids , which enables feature learning . Li et al. , generalize the convolution operation on point-clouds and employ X -transformations to learn the weights and permutations for the features ( Li et al. , 2018 ) . Through this , the proposed PointCNN leverages spatial-local correlations of point clouds , irrespective of the order of the input . Notably , although these architectures can learn spatial features of point-clouds , they are designed to work with static data , thus have limited ability to discover temporal dependencies . 3 CONVOLUTIONAL POINT-CLOUD LSTM . Next , we describe in detail the concept and properties of forecasting over point cloud-streams . We then introduce the DConv operator , which is at the core of our proposed CloudLSTM architecture . Finally , we present CloudLSTM and its variants , and explain how to combine CloudLSTM with Seq2seq learning and attention mechanisms , to achieve precise forecasting over point-cloud streams . 3.1 FORECASTING OVER POINT-CLOUD STREAMS . We formally define a point-cloud containing a set of N points , as S = { p1 , p2 , · · · , pN } . Each point pn ∈ S contains two sets of features , i.e. , pn = { νn , ςn } , where νn = { v1n , · · · , vHn } are value features ( e.g. , mobile traffic measurements , air quality indexes , etc . ) of pn , and ςn = { c1n , · · · , cLn } are its L-dimensional coordinates . At each time step t , we may obtain U different channels of S by conducting different measurements denoted by Sυt = { S1t , · · · , SUt } , Sυt ∈ RU×N× ( H+L ) . Here , different U resemble the RGB channels in images . We can then formulate the J-step point-cloud stream forecasting problem , given M observations , as : Ŝυt+1 , · · · , Ŝυt+J = argmax Sυt+1 , ··· , Sυt+J p ( Sυt+1 , · · · , Sυt+J |Sυt , · · · , Sυt−M+1 ) . ( 1 ) Note that , in some cases , each point ’ s coordinates may be unchanged , since the data sources are deployed at fixed locations . An ideal point-cloud stream forecasting model should embrace five key properties , similar to other point-cloud applications and spatiotemporal forecasting problems ( Qi et al. , 2017a ; Shi et al. , 2017 ) : ( i ) Order invariance : A point cloud is usually arranged without a specific order . Permutations of the input points should not affect the output of the forecasting ( Qi et al. , 2017a ) . ( ii ) Information intactness : The output of the model should have exactly the same number of points as the input , without losing any information , i.e. , Nout = Nin . ( iii ) Interaction among points : Points in S are not isolated , thus the model should be able to capture local dependencies among neighboring points and allow interactions ( Qi et al. , 2017a ) . ( iv ) Robustness to transformations : The model should be robust to correlation-preserving transformation operations on point-clouds , e.g. , scaling and shifting ( Qi et al. , 2017a ) . ( v ) Location variance : The spatial correlations among points may change over time . Such dynamic correlations should be revised and learnable during training ( Shi et al. , 2017 ) . In what follows , we introduce the Dynamic Point Cloud Convolution ( DConv ) operator as the core module of the CloudLSTM , and explain how DConv satisfies the aforementioned properties . 3.2 DYNAMIC CONVOLUTION OVER POINT CLOUD . The Dynamic Point Cloud Convolution operator ( DConv ) generalizes the ordinary convolution on grids . Instead of computing the weighted summation over a small receptive field for each anchor point , DConv does so on point-clouds , while inheriting desirable properties of the ordinary convolution operation . The vanilla convolution takes Uin channels of 2D tensors as input , and outputs Uout channels of 2D tensors of smaller size ( if without padding ) . Similarly , the DConv takes Uin channels of a point-cloud S , and outputs Uout channels of a point-cloud , but with the same number of elements as the input , to ensure the information intactness property ( ii ) discussed previously . For simplicity , we denote the ith channel of the input set as Siin and the jth channel of the output as S j out . Both Siin and Sjout are 3D tensors , of shape ( N , ( H+L ) , Uin ) and ( N , ( H+L ) , Uout ) respectively . We also define QKn as a subset of points in Siin , which includes the K nearest points with respect to pn in the Euclidean space , i.e. , QKn = { p1n , · · · , pkn , · · · , pKn } , where pkn is the k-th nearest point to pn in the set Siin . Note that pn itself is included in QKn as an anchor point , i.e. , pn ≡ p1n . Recall that each pn ∈ S contains H value features and L coordinate features , i.e. , pn = { νn , ςn } , where νn = { v1n , · · · , vHn } and ςn = { c1n , · · · , cLn } . Similar to the vanilla convolution operator , for each pn 0 1 1 0 1 1 Figure 2 : Illustration of the DConv operator , with a single input channel and K = 6 neighbors . For every p ∈ S1in , DConv weights its K neighboring set QKn = { p1n , · · · , p6n } to produce values and coordinate features for p′n ∈ S1out . Here , each wk is a set of weights w with index k ( i.e. , k-th nearest neighbor ) in Eq . 2 , shared across different p. in Siin , the DConv sums the element-wise product over all features and points in QKn , to obtain the values and coordinates of a point p′n in S j out . Note that we assume the value features are related to their positions at the previous layer/state , to better exploit the dynamic spatial correlations . Therefore , we aggregate coordinate features c ( pkn ) l i when computing the value features v h′ n , j . The mathematical expression of the DConv is thus : vh ′ n , j = ∑ i∈Uin ∑ pkn∈QKn ( ∑ h∈H wh , h ′ , k i , j v ( p k n ) h i + ∑ l∈L w ( H+l ) , h′ , k i , j c ( p k n ) l i ) + bj , cl ′ n , j = σ ( ∑ i∈Uin ∑ pkn∈QKn ( ∑ h∈H wh , l ′ , k i , j v ( p k n ) h i + ∑ l∈L w ( H+l ) , l′ , k i , j c ( p k n ) l i ) + bj ) , Sjout = ( p′1 , · · · , p′N ) ( 2 ) = ( ( ( v1 ′ 1 , · · · , vH ′ 1 ) , ( c 1′ 1 , · · · , cL ′ 1 ) ) , · · · , ( ( v1 ′ N , · · · , vH ′ N ) , ( c 1′ N , · · · , cL ′ N ) ) ) . In the above , we define learnable weights W as 5D tensors with shape ( Uin , K , ( H + L ) , ( H + L ) , Uout ) . The weights are shared across different anchor points in the input map . Each element wm , m ′ , k i , j ∈ W is a scalar weight for the i-th input channel , j-th output channel , k-th nearest neighbor of each point corresponding to the m-th value and coordinate features for each input point , and m′-th value and coordinate features for output points . Similar to the convolution operator , we define bj as a bias for the j-th output map . In the above , h and h′ are the h ( ′ ) -th value features of the input/output point set . Likewise , l and l′ are the l ( ′ ) -th coordinate features of the input/output . σ ( · ) is the sigmoid function , which limits the range of predicted coordinates to ( 0 , 1 ) , to avoid outliers . Before feeding them to the model , the coordinates of raw point-clouds are normalized to ( 0 , 1 ) by ς = ( ς − ςmin ) / ( ςmax − ςmin ) , on each dimension . This improves the transformation robustness of the operator . The K nearest points can vary for each channel at each location , because the channels in the pointcloud dataset may represent different types of measurements . For example , channels in the mobile traffic dataset are related to the traffic consumption of different mobile apps , while those in the air quality dataset are different air quality indicators ( SO2 , CO , etc. ) . The spatial correlations will vary between different measurements ( channels ) , due to human mobility . For instance , more people may use Facebook at a social event , but YouTube traffic may be less significant in this case . This will be reflected by the data consumption of each app . The same applies to air quality indicators affected by vehicle movement and factory working times . We want these spatial correlations to be learnable , so we do not fix the K nearest neighbors across channels , but encourage each channel to find the best neighbor set . This is also a contribution of the CloudLSTM , which helps improve the forecasting performance . We provide a graphical illustration of DConv in Fig . 2 . For each point pn , the DConv operator weights its K nearest neighbors across all features , to produce the values and coordinates in the next layer . Since the permutation of the input neither affects the neighboring information nor the ranking of their distances for any QKn , DConv is a symmetric function whose output does not depend on the input order . This means that the property ( i ) discussed in Sec . 3.1 is satisfied . Further , DConv is performed on every point in set Siin and produces exactly the same number of features and points for its output ; property ( ii ) is therefore naturally fulfilled . In addition , operating over a neighboring point set , irrespective of its layout , allows to capture local dependencies and improve the robustness to global transformations ( e.g. , shifting and scaling ) . The normalization over the coordinate features further improves the robustness to those transformations , as shown by the proof in Appendix C. This enables to meet the desired properties ( iii ) and ( iv ) . More importantly , DConv learns the layout and topology of the cloud-point for the next layer , which changes the neighboring set QKn for each point at output Sjout . This enables the “ location-variance ” ( property ( v ) ) , allowing the model to perform dynamic positioning tailored to each channel and time step . This is essential in spatiotemporal forecasting neural models , as spatial correlations change over time ( Shi et al. , 2017 ) . DConv can be efficiently implemented using simple 2D convolution , by reshaping the input map and weight tensor , which can be parallelized easily in existing deep learning frameworks . We detail this in Appendix A and provide a complexity analysis of the DConv operator in Appendix B . Relations with PointCNN ( Li et al. , 2018 ) and Deformable Convolution ( Dai et al. , 2017 ) . The DConv operator builds upon the PointCNN ( Li et al. , 2018 ) and deformable convolution neural network ( DefCNN ) on grids ( Dai et al. , 2017 ) , but introduces several variations tailored to pointcloud structural data . PointCNN employs the X -transformation over point clouds , to learn the weight and permutation on a local point set using multilayer perceptrons ( MLPs ) , which introduces extra complexity . This operator guarantees the order invariance property , but leads to information loss , since it performs aggregation over points . In our DConv operator , the permutation is maintained by aligning the weight of the ranking of distances between point pn and QKn . Since the distance ranking is unrelated to the order of the inputs , the order invariance is ensured in a parameter-free manner without extra complexity and loss of information . Further , the DConv operator can be viewed as the DefCNN ( Dai et al. , 2017 ) over point-clouds , with the differences that ( i ) DefCNN deforms weighted filters , while DConv deforms the input maps ; and ( ii ) DefCNN employs bilinear interpolation over input maps with a set of continuous offsets , while DConv instead selects K neighboring points for its operations . Both DefCNN and DConv have transformation modeling flexibility , allowing adaptive receptive fields on convolution .

Summary: The paper proposed a new convolution operator, named dynamic post-cloud convention over spatiotemporal data, and the convolution operator can be embedded in different neural network architectures, like recurrent neural networks. In order to achieve the convolution over point-clouds by using both value features and the spatial-features, given a data point, the convolution is conducted over its k-nearest neighbors generated by CNN.  They compared the proposed convolution method by embedding it into RNN, GRN, and LSTM against a number of existing methods on two datasets, in terms of MAE, RMSE, PSNR, and SSIM. Overall, this paper is interesting but needs some clarifications on