Deep Convolution for Irregularly Sampled Temporal Point Clouds

1 INTRODUCTION . Many real-world problems feature observations that are sparse and irregularly sampled in both space and time . Weather stations scattered across the landscape reporting at variable rates without synchronization ; citizen-science applications producing observations at the whim of individuals ; or even opportunistic reports of unit positions in search-and-rescue or military operations . These sparse and irregular observations naturally map to a set of discrete space-time points – forming a spatiotemporal point cloud representing the underlying process . Critically , the dynamics of these points are often highly related to the other points in their spatio-temporal neighborhood . Modelling spatio-temporal point clouds is difficult with standard deep networks which assume observations are dense and regular – at every grid location in CNNs , every time step in RNNs , or both for spatio-temporal models like Convolutional LSTMs ( Xingjian et al. , 2015 ) . While there has been work examining irregularly sampled data through time ( Rubanova et al. , 2019 ; Shukla & Marlin , 2018 ) and in space ( Wu et al. , 2019 ) , modeling both simultaneously has received little attention ( Choy et al. , 2019 ) . This is due in part to the difficulty of scaling prior solutions across both space and time . For instance , voxelization followed by sparse convolution ( Choy et al. , 2019 ) or dense imputation ( Shukla & Marlin , 2018 ) now face a multiplicative increase in the number of cells . Rather than forcing irregular data into dense representations , an emerging line of research treats spatial point-clouds as first-class citizens ( Qi et al. , 2017a ; b ; Su et al. , 2018 ; Xu et al. , 2018 ) . Several works directly extend 2D convolutions to point clouds ( Simonovsky & Komodakis , 2017 ; Wang et al. , 2019 ; Hermosilla et al. , 2018 ) , with ( Wu et al. , 2019 ) being the first that allows efficient exact computation of convolution with dozens of layers . In this work , we build on this line of research to model spatio-temporal point clouds . Specifically , we extend the work of Wu et al . ( 2019 ) with an additional module to reason about point representations through time . Our new model , TemporalPointConv ( TPC ) , is a simple but powerful extension that can learn from an arbitrary number of space-time points . Each layer in TemporalPointConv updates the representation of each point by applying two operators in sequence – one that considers the spatial neighborhood in a narrow temporal window and another that models how this spatial representation changes over time . By factorizing the representation update into separate spatial and temporal operators , we gain significant modeling flexibility . Further , by operating directly on point clouds , we can predict observations at arbitrary space-time , regardless of the distribution of observations . We demonstrate TemporalPointConv on two distinct problems : 1 ) predicting future states of a custom Starcraft II environment involving battles between variable-sized groups , and 2 ) predicting the weather at stations distributed throughout the state of Oklahoma . Further , we show the utility of these networks in identifying damaged or anomalous weather sensors after being trained exclusively on the associated prediction problem . The results show that TemporalPointConv outperforms both state of the art set functions and a discrete sparse convolution algorithm in terms of raw performance , ability to detect anomalies , and generalization to previously unseen input and query distributions . 2 RELATED WORK . Xingjian et al . ( 2015 ) gives an early approach to spatio-temporal modeling via convolution by incorporating a standard convolutional structure into the latent memory of an LSTM . This approach is appropriate for situations where the data is regularly sampled in both space and time , which is different from our setting . Interaction networks ( Battaglia et al. , 2016 ) and related approaches allow for modeling sets of interacting objects or points over time , with an original motivation to model physics processes . These models are more flexible in their modeling of spatial relationships among points . However , there is an assumption of uniform temporal sampling , which is violated in our setting . A significant amount of work on spatio-temporal modeling for non-uniform spatial sampling uses Graph Convolutional Networks ( GCNs ) for modeling spatial interactions . For example , Li et al . ( 2018b ) used a GCN followed by an RNN and Yu et al . ( 2018 ) used GCNs for spatial correlation and temporal convolution for temporal correlations . They require sampling at continuous temporal intervals and did not deal with generalization outside the fixed given graph . Rather , our approach generalizes to any spatio-temporal point outside of the training data . Yao et al . ( 2019 ) introduces an attention model to deal with dynamic spatial relationships , however this is only possible for the dense CNN version in their paper , whereas their version with irregular spatial sampling utilizes the GCN and shares the same issues with the above GCN approaches . PointNet ( Qi et al. , 2017a ) sparked significant interest in networks for 3D Point cloud processing . A number of networks have been proposed ( Qi et al. , 2017a ; b ; Su et al. , 2018 ; Xu et al. , 2018 ) with the highest performing using either sparse convolutional networks ( Graham & van der Maaten , 2018 ; Choy et al. , 2019 ) or point convolutional networks ( Wu et al. , 2019 ; Thomas et al. , 2019 ) . Set networks , such as DeepSets ( Zaheer et al. , 2017b ) , are similar to PointNet ( Qi et al. , 2017a ) with neither explicitly considering neighborhood information of elements/points , making them less powerful than convolutional methods . Recently , Horn et al . ( 2020 ) proposed a set network approach for non-uniform time-series prediction , which encodes time into the feature vector of points . Our experiments show that this approach is outperformed by our convolutional method . Sparse convolutional networks are similar to dense volumetric convolutional networks that use a regular grid to discretize space-time , but they are only computed at locations with occupied points . Minkowski networks ( Choy et al. , 2019 ) is a sparse convolutional network that models spatio- temporal correlations by concatentating the spatial location and time for each point sample into a 4D tesseract . It is thus sensitive to an appropriate resolution for the discretization since excess sparsity can result in empty neighborhoods and trivial convolutions , and too coarse a resolution may result in an inaccurate representation of the data . Furthermore , the approach has difficulties accounting for the case where points should be treated as moving entities themselves . On the other hand , point convolutions discretize 3D volumetric convolutions on each point directly and hence easily generalize to the entire space under irregular sampling density . Early versions ( Simonovsky & Komodakis , 2017 ; Hermosilla et al. , 2018 ; Wang et al. , 2019 ) require explicit discretization hence can not scale to large networks . Recently , PointConv ( Wu et al. , 2019 ) proposes an equivalent form that avoids explicit discretization and significantly improved scalability . However , so far it has been applied only to static point clouds . Our work builds on PointConv , by extending it in the temporal direction and demonstrating that space-time convolutions can be effectively learned and used for modeling and anomaly detection . On the temporal side , a significant amount of recent state-of-the-art were based on point processes which studies time series models from a statistical perspective ( Du et al. , 2016 ; Li et al. , 2018a ; Zuo et al. , 2020 ; Zhang et al. , 2019 ) . These support irregular temporal sampling , but generally do not consider the spatial correlation among points . 3 PROBLEM SETUP . We consider extrapolative tasks in which the value at new locations must be inferred from existing observations . Let P be a spatio-temporal point cloud with each individual point pj ∈ P defined as pj = ( lj , tj , oj ) where pj exists at location lj at time tj and has associated features oj ( e.g . temperature and humidity values for a weather station ) . Further , let Q be a set of query locations at which the model is to make predictions given P . For example , a forecasting model might be given queries qk = ( lk , tk ) for locations in the future and be tasked with predicting the corresponding features ok representing the desired properties to be predicted . We place no restrictions on the regularity of either P or Q such that this corresponds to a setting where both input and output may be sparse and irregularly sampled through space and time . Further , query points may be in the future , the past , or concurrent with those in P – corresponding to weather forecasting , backcasting , or nowcasting respectively . We aim to train models that can accurately answer queries as represented via training set of point-cloud / query-set pairs D = { ( Pi , Qi ) } Ni=1 . 4 TEMPORAL POINTCONV ARCHITECTURE . Given a spatio-temporal point-cloud containing points pj = ( lj , tj , oj ) , a Temporal PointConv layer is an operator that produces an updated point representation p′j = ( lj , tj , o ′ j ) for each point . The updated feature representation o′j incorporates information from a spatio-temporal neighborhood around pj . This is accomplished by applying two point-based convolutional operators in sequence for each point – first a spatial PointConv over points within a narrow temporal band , and then a temporal PointConv over points within a narrow spatial band . These Temporal PointConv layers can be stacked to arbitrary depth . Below we give background on PointConv and describe our model . 4.1 PRELIMINARIES : POINTCONV . PointConv is based on the idea of discretizing continuous convolution on irregularly sampled points : Conv ( P , p0 ; w , d ( · , · ) ) = ∑ pi∈Nd ( p0 ) 〈w ( pi − p0 ) , oi〉 ( 1 ) where P is a point cloud with features at each point , w ( · ) is a vector-valued weight function of the positional difference between a point pi in the neighborhood Nd of a centroid p0 , defined by a metric d , and oi is the input features at pi . w ( · ) can be learned with a neural network ( Simonovsky & Komodakis , 2017 ) . PointConv ( Wu et al. , 2019 ) introduces an equivalent form so that w does not need to be computed explicitly , saving computation and memory . This approach is flexible since w ( · ) as a function can apply to any point in the space of P , hence convolution can be computed over any irregularly sampled neighborhoodNd . We note that this even holds when we did not have any feature at p0 , since a neighborhood can still be found even in this case and eq . ( 1 ) can still be used . Previously , PointConv has only been used in spatial domains in cases where p0 has features associated with it . In this paper we generalize it to spatio-temporal neighborhoods and to p0 that are featureless query points . For expositional clarity , we denote PointConv as an operator that transforms a feature-augmented point-cloud P into a new point-cloud P ′ consisting of points at target locations Q with eq . ( 1 ) : P ′ = PointConv ( P , Q ; d ( · , · ) ) , where we will omit Q if Q = P .

Summary: This paper studies the problem of modeling spatial-temporal point clouds which are sampled at irregular space and time points. It proposes the Temporal PointConv model which is an extension of the PointConv model (Wu et al., 2019). In particular, PointConv computes a convolution by aggregating the features of nearby points of a point p as the new feature of p.  Temporal PointConv extends this by aggregating the features of points near p in both space and time in a two-step process: first weighting the aggregation by the space distance and then weighting the aggregation by the temporal distance.