Deep Convolution for Irregularly Sampled Temporal Point Clouds
Authors: Erich Merrill III
Stefan Lee
Li Fuxin
Thomas G Dietterich
Alan Fern
Venue:  (2021)
Keywords: point cloud
convolution
irregular sampling
starcraft
nowcasting

Abstract: We consider the problem of modeling the dynamics of continuous spatial-temporal processes represented by irregular samples through both space and time. Such processes occur in sensor networks, citizen science, multi-robot systems, and many others. 
We propose a new deep model that is able to directly learn and predict over this irregularly sampled data, without voxelization, by leveraging a recent convolutional architecture for static point clouds. The model also easily incorporates the notion of multiple entities in the process. In particular, the model can flexibly answer prediction queries about arbitrary space-time points for different entities regardless of the distribution of the training or test-time data. We present experiments on real-world weather station data and battles between large armies in StarCraft II. The results demonstrate the model's flexibility in answering a variety of query types and demonstrate improved performance and efficiency compared to state-of-the-art baselines. 

Under review as a conference paper at ICLR 2021
DEEP CONVOLUTION FOR IRREGULARLY SAMPLED TEMPORAL POINT CLOUDS
Anonymous authors Paper under double-blind review
ABSTRACT
We consider the problem of modeling the dynamics of continuous processes represented by irregular samples through both space and time. Such pro- cesses occur in sensor networks, citizen science, multi-robot systems, and many others. We propose a new deep model that able to directly learn and predict over this irregularly sampled data, without voxelization, by leveraging recent convolutional architecture for static point clouds. The model also easily incorpo- rates the notion of multiple entities in the process. In particular, the model flexibly answer prediction queries about arbitrary space- time points for different entities regardless of the distribution of the training test-time data. We present experiments on real-world weather station data and battles between large armies in StarCraft II. The results demonstrate the model's 's flexibility in answering variety of query types and demonstrate improved performance and efficiency compared to state-of-the-an baselines.
INTRODUCTION
Many real-world problems feature observations that are sparse and irregularly sampled in both space and time. Weather stations scattered across the landscape reporting at variable rates without synchro- nization; citizen-scienee applications producing observations the whim of individuals; or even opportunistic reports of unit positions in cearch-and-rescus or military operations. These sparse and irregular observations naturally map set of discrete space-time time points forming a spatio- temporal point cloud representing the underlying process. Critically, the dynamics of these points are often highly related the other points their spatio-temporal neighborhood.
Modelling spatio-temporal point clouds is difficult with standard deep networks which assume ob- servations are dense and regular - at every grid location in CNNs, every time step in RNNs, or both for spatio-temporal models like Convolutional LSTMs (Xingjian al.) (015). While there has been work examining irregularly sampled data through time (Rubanova et al. 2019 Shukla Marlin 2018) and in space (Wu et al.] (019). modeling both simultaneously has received little attention (Choy et al. 2019). This is due in part to the difficulty of scaling prior solutions across both space and time. For instance, voxelization followed by sparse convolution (Choy al. 2019) dense imputation (Shukla & Marlin 2018) now face a multiplicative increase in the number of cells.
Rather than foreing irregular data into dense representations, an emerging line of research treats spatial point-clouds as first-class citizens (Qi et al. 2017a Su al. 2018)) 8 et al. 2018). Several Komodakis 2017 Wang et al. 2019 Hermosilla et al. 2018). with (Wu et al. 2019) being the first that allows efficient exact computation of convolution with dozens of layers. In this work, build on this line of research to model spatio-temporal point clouds. Specifically, we extend the work of WW et al. (2019) with an additional module to reason about point representations through time.
works directly extend 2D convolutions to point clouds (Simonovsky
Our new model, TemporalPointConv (TPC). is simple but powerful extension that can learn from an arbitrary number of space-time points. Each layer in TemporalPointConv updates the representa- tion of each point by applying two operators in sequence one that considers the spatial neighbor- hood in a narrow temporal window and another that models how this spatial representation changes over time. By factorizing the representation update into separate spatial and temporal operators, gain significant modeling flexibility. Further, by operating directly on point clouds, we can predict observations arbitrary space-time, regardless of the distribution of observations.

---

Under review as a conference paper at ICLR 2021
Festure
Fostion
PointConv (Temporal)
PointConv (Spatial)
Temporal Pointomm Layor
PoinsConv (Spatial)
b
q
PointConv (Temporal)
tengs
Toct Come
of
Updateed Feature
Podioo
Figure I: TemporalPointCon operates on unsynchronized sets of spatio-temporal samples by ap- plying two point-based convolutional operators in sequence, each of which exploits separate notions of either spatial temporal locality.
We demonstrate TemporalPointCon two distinct problems: 1) predicting future states of cus- tom Starcraft II environment involving battles between variable-sized groups, and 2) predicting the weather at stations distributed throughout the state of Oklahoma. Further, show the utility of these networks identifying damaged or anomalous weather sensors after being trained exclusively on the associated prediction problem. The results show that TemporalPointConv outperforms both state of the art set functions and a discrete sparse convolution algorithm in terms of raw performance, ability detect anomalies, and generalization previously unseen input and query distributions.
2 RELATED WORK
Xingjian et (2015) gives an early approach to spatio-temporal modeling via convolution by in- corporating a standard convolutional structure into the latent memory of an LSTM. This approach appropriate for situations where the data is regularly sampled in both space and time, which is different from our setting. Interaction networks (Battaglia et al.) (016) and related approaches allow for modeling sets of interacting objects or points over time, with an original motivation to model physics processes. These models are more flexible in their modeling of spatial relationships among points. However, there is an assumption of uniform temporal sampling, which is violated in our setting.
A significant amount of work on spatio-temporal modeling for non-uniform spatial sampling uses Graph Convolutional Networks (GCNs) for modeling spatial interactions. For example, Li et al. (2018b) used a GCN followed by an RNN and Yu et al. (2018) used GCNs for spatial correlation and temporal convolution for temporal correlations. They require sampling at continuous temporal intervals and did not deal with generalization outside the fixed given graph. Rather, our approach generalizes any spatio-temporal point outside of the training data. Yao et al. (2019) introduces an attention model to deal with dynamic spatial relationships, however this is only possible for the dense CNN version in their paper, whereas their version with irregular spatial sampling utilizes the GCN and shares the same issues with the above GCN approaches.
PointNet (Qi et al. 2017a) sparked significant interest in networks for 3D Point cloud processing.
A number of networks have been proposed (Qi et al. 2017a by Su et al. 2018 Xu et al. 2018
with the highest performing using either sparse convolutional networks (Graham & van der Maaten
2018 Choy et al. 2019) or point convolutional networks (Wu et al. 2019 Thomas et al. 2019).
Set networks, such as DeepSets [Zaheer et al. 2017b), are similar to PointNet (Qi et al. 2017a) with neither explicitly considering neighborhood information of clements/points, making them less powerful than convolutional methods. Recently, [orm et al. (2020) proposed a set network approach for non-uniform time-series prediction, which encodes time into the feature vector of points. Our experiments show that this approach is outperformed by our convolutional method.
Sparse convolutional networks similar to dense volumetric convolutional networks that regular grid to discretize space-time, but they are only computed at locations with occupied points. Minkowski networks (Choy et al., 2019) sparse convolutional network that models spatio-
2

---

Under review as a conference paper at ICLR 2021
temporal correlations by concatentating the spatial location and time for each point sample into 4D tesseract. is thus sensitive to an appropriate resolution for the discretization since excess sparsity can result in empty neighborhoods and trivial convolutions, and too coarse resolution may result in an inaccurate representation of the data. Furthermore, the approach has difficulties accounting for the case where points should be treated as moving entities themselves.
On the other hand, point convolutions discretize 3D volumetric convolutions on each point directly and hence easily generalize to the entire space under irregular sampling density. Early versions (Si-) monovsky & Komodakis) 2017; Hermosilla et al. 2018 Wang et al. 2019) require explicit dis- cretization hence cannot scale to large networks. Recently, PointConv (Wu al.) 2019) proposes an equivalent form that avoids explicit discretization and significantly improved scalability. However, so far has been applied only to static point clouds. Our work builds on PointConv, by extending in the temporal direction and demonstrating that space-time convolutions can be effectively learned and used for modeling and anomaly detection.
On the temporal side, significant amount of recent state-of-the-art were based on point processes which studies time series models from a statistical perspective (Du et al. 2016 Li et al., 2018a| Zuo et al. 2020 Zhang et al. 2019). These support irregular temporal sampling, but generally do not consider the spatial correlation among points.
PROBLEM SETUP
We consider extrapolative tasks which the value at new locations must inferred from existing observations. Let P be a spatio-temporal point cloud with each individual point Py E P defined as Py = (4 tj. 0,) where Py exists at location I, at time ty and has associated features o, (e.g. temperature and humidity values for weather station). Further, let Q be a set of query locations at which the model is to make predictions given P. For example, a forecasting model might be given queries 9: (4k,t)) for locations in the future and be tasked with predicting the corresponding features representing the desired properties to be predicted. We place no restrictions on the regularity of either P or Q such that this corresponds be sparse and irregularly sampled through space and time. Further, query points may be in the future, the past, or concurrent with those in corresponding to weather forecasting, backcasting, or nowcasting respectively. We aim to train models that can accurately answer queries as represented via training set of point-cloud / query-set pairs = [(P,. Q.)
setting where both input and output may
TEMPORAL POINTCONV ARCHITECTURE
Given a spatio-temporal point-cloud containing points p, = (1y, ty. 0;), a Temporal PointCony layer is an operator that produces an updated point representation pl, = (1g ty. oy) for each point. The updated feature representation of incorporates information from a spatio-temporal neighborhood around Py- This is accomplished by applying two point-based convolutional operators in sequence for each point- - first a spatial PointConv over points within narrow temporal band, and then temporal PointConv over points within narrow spatial band. These Temporal PointConv layers can be stacked to arbitrary depth. Below we give background on PointConv and describe model.
4.1 PRELIMINARIES: POINTCONV
PointConv is based on the idea of discretizing continuous convolution on irregularly sampled points:
Conv(P, Po; w, d(-,))) = Σ
(w(p: - PP),
(I)
P:ENs(Po)
where P is a point cloud with features at each point, w(-) vector-valued weight function of the positional difference between a point p, in the neighborhood Na of centroid Po, defined by metric d, and o, is the input features at p,. w(-) can be learned with a neural network (Simonovsky & Komodakis 2017). PointConv (Wu et al. 2019) introduces equivalent form that does not need to be computed explicitly, saving computation and memory.
This approach is flexible since w(.) convolution can be computed over any irregularly sampled neighborhood Ng, We note that this even holds when we did not have any feature at po: since neighborhood can still be found even in this
function can apply to any point in the space of P. hence
3

---

Under review as a conference paper at ICLR 2021
case and eq. can still be used. Previously, PointConv has only been used in spatial domains in cases where Po has features associated with it. In this paper we generalize it to neighborhoods and to Po that are featureless query points.
For expositional clarity, we denote PointConv as an operator that transforms feature-augmenter point-cloud P into a new point-cloud P' consisting of points at target locations with eq. [ P' = PointtCon((P, Q; :)), where we will omit Q if Q P.
4.2 TEMPORAL POINTCONY
Given a spatio-temporal point-cloud P... ((g. Lg, 0j 11)) and set of queries Q the Temporal PointConv operations considers the relative position from each query to the elements Pin and their representative features to produce a of predictions X corresponding to the query set Q.
Spatial Convolution. First, each point's feature is updated based on the spatial neighborhood of temporally co-occurring points. However, the points may be irregularly spaced in time, there may be no points that precisely co-occur. We instead consider those in a fixed window of time. Thanks to the flexibility PointConv operations, describe this by defining the piece-wise distance function:
f114 4 L ll2
if |t, 144-t) tyl S Et. otherwise
dspatial (Pi: P5)
We then apply a PointConv operator to update features: Pspatial = TointCCnmo(Pae Pini depatiat), where each point in P Psppatiat has updated feature (1i, 4,, (4,))) (
Temporal Convolution. We then perform an analogous operation through time. We would like consider the past and future of each point; however, this requires determining correspondence between points through time. the underlying point-cloud represents static points such as weather stations, this can simply be based on a small spatial window. If the points correspond known entities that are moving, we instead assume tracking and can use those entity labels to determine temporal neighborhoods each consisting exclusively of single entity's samples throughout time. For clarity, we present the distance function for the first case below:
diemporal (p: PD) 4 to lla if 1| 1
1 ll2 S 6,
otherwise
Before applying the temporal PointConv, we first apply residual connection for each point, con- catenating the input and spatial features. We denote this as Pres {(l,, ty. 1of of' j} where [] denotes concatenation. As before, we apply a PointConv operator with kernels de- fined only over differences in time as: Ptemporal PointConuuPPP.. (remporat ()) where
Ptemporal {11 f ( ().))
Combined Representation. To compute the final output point-cloud, we concatenate the original, spatial, and temporal representations and transform them through an MLP such that
(s) (fmp))) [s}. Pout {lg,,t,, oo .0,
(4)
We denote multiple stacked layers via p(d+1)
PointConoi ((()).
4.3 EXTRAPOLATING TO NEW POINTS
After applying one or more layers of Temporal PointCony as described above, we apply one final query PointConv to the latent spatio-temporal point cloud Pout resulting from this encoding process. For this, we define new problem-dependen query distance function d' which could be dspatial, diemporal, or a combination of both. This enables us to calculate a corresponding latent feature y for the each query point.
Y PointConv(PPeee Pous, Q; dquery(:
(5)
Finally, we apply an MLP g to transform each latent query representation into a final predictions X (lv E Y} corresponding the set of queries Q.
4

---

Under review as a conference paper at ICLR 2021
5 EXPERIMENTS
We consider two problem domains for our experiments which we describe below.
Starcraft II. To evaluate TemporalPointCony on entity-based dynamics, designed a custom Star- craft II scenario in which two opposing armies consisting of random numbers of three distinct unit types are created and then fight featureless battlefield. Each episode is allowed to without any external influence until one team has been eliminated or the time limit expires. This allows us to learn the dynamics of a battle between a large group of units without any confounding factors such as player inputs. We use the PySC2 library (Vinyals et al. 2017) to record regular observations of the game state as each episode plays out.
We use these regularly sampled episode histories to generate individual training examples. Specifi- cally, we select 'reference timestep' within the episode, sample set of 'history offsets' H from a provided history distribution, and a set of "query offsets' R from a provided query distribution. We collect unit properties corresponding to these sampled relative time steps to serve as point features. We determine the prediction targets with the same procedure using the sampled query officts. This procedure is used to sample an arbitrary number of training examples from the set of episode his- tories by varying the reference timestep and remaall the history and query offsets as desired. Following this procedure on our dataset of 92,802 episodes yields 2.5 million training examples.
We define the 'property loss' for unit state prediction the sum of the mean squared error of each of the unit's predicted numeric properties (i.e. health, shields, position) and the cross entropy loss of the unit's predicted categorical properties (i.e. orientation). Similarly, the 'alive loss' is the cross entropy loss between the network's 's alive/dead prediction values and a flag indicating if the unit was present and alive in the given timestep. We then define the total loss for set of unit state predictions as the sum of the alive loss for all units and with property loss for every unit that is actually alive the given timesteps. This additional condition is necessary due the fact that dead units do not have recorded properties we can use to determine property loss.
As PySC2 assigns a unique, consistent ID to each unit which provides perfect tracking across all timesteps, we use an entity-based temporal distance function when instantiating the query PointConv layer for this problem as described in section 4.2 above.
Weather Nowcasting. To evaluate the ability of the TemporalPointConv architecture to reason about spatio-temporal dynamics, we derive weather nowcasting problems from dataset of weather conditions as recorded by weather stations throughout Oklahoma. The original dataset consists of weather sensor readings from each weather station every five minutes throughout the entirety of the year 2008, associated quality metrics for each sensor in each reading, and metadata about each weather station such as its position and local soil properties.
10% of the weather stations are randomly selected to be held out as test stations and excluded from the training process, while the remaining 90% are used to generate problems for training. We derive training problems from the larger dataset by selecting time point and randomly selecting 10% of the remaining training stations to be targets. All non-target training station readings and their associated station metadata within the hour preceding 1. collected input weather data. Any sample within the collected data with an associated quality metric indicating a malfunctioning or missing sensor is discarded. Furthermore, we randomly discard an additional 20% of the remaining samples to decrease the level of time synchronization in the input. Following this procedure on our dataset of weather sensor readings results in over 14,000 training examples.
The model is then tasked with predicting weather properties at time t for each of the target stations using the provided input data from the preceding hour. Specifically, the networks are evaluated on their ability to predict the relative humidity, air temperature, air pressure, and wind speed at each specified target location. We define the prediction loss as the sum of the mean square error between the network's prediction for each of these properties and the actual recorded values. Due to the large difference in magnitudes between these readings, normalize each prediction and target measurement value such that the 10th percentile value to 90th percentile value of that measurement within the entire dataset is mapped to the range [0, 1]. This prevents the training process from naturally favoring measurements with much higher average magnitude than the others.
As our queries for this problem are purely spatial, we use the spatial distance function eq. (2) as the query distance function when instantiating the query PointConv layer for this problem.
5

---

Under review as a conference paper at ICLR 2021
DeepSets
3x10
10
Mink
TPC
Loss
10a
Loss
x010
10-1
10-2
109
0.6
Epoch
Epoch
(a) Weather Domain
(b) Starcraft Domain
Figure 2: Validation performance throughout the training process averaged across four runs.
5.1 BASELINE IMPLEMENTATIONS
Set Functions for Time Series & DeepSets. Our Temporal PointConv architecture leverages Point- Conv as convolution-eque equivalent set function. We can evaluate this choice by replacing each Point- Conv module with different set function, such as DeepSets (Zaheer et al. (017a) or Set Functions for Time Series (SeFT) (Horn et al. 2020), Whereas PointConv takes as input set of point loca- tions and a set of point features, SeFT and DeepSets only consume a single set of features. How- ever, the neighborhood and distance function mechanisms introduced for Temporal PointConv can still be applied. Therefore, evaluate the other set functions by simply replacing each instance of PointCona P) with SeFT([44, li oi]li)) or DeepSels(1))) 4i, o(]li))
Minkowski Networks. We evaluate Minkowski networks (Choy et al. 2019) replacing each spatial-temporal PointConv step with a Minkowski convolution layer that operates on the combined spatio-temporal vector space inhabited by the raw input samples. This necessarily requires dis- cretizing said vector space into a sparse voxel grid. We choose voxel resolution of 6km for the weather domain, and 0.05 in game units for the starcraft domain. We VVdia's MinkowskiEngine codebase to provide the Minkowski convolution implementation.
We trained Temporal PointConv (TPC), Set Function for Time Series (ScFT), DeepSets, and Minkowski networks instantiated with the hyperparameter settings described in appendix B on both the Starcraft and weather nowcasting domains. For the Stareraft II domain, models were trained for one epoch (owing to the massive size of the generated Starcraft II dataset), whereas for weather nowcasting they were trained for 24 epochs. All networks were trained with a cosine learning rate decay with warm restarts configured such that the learning rate cycles from its maximum value to its minimum three times throughout each training run.
5.2 RESULTS
Dynamics Prediction Accuracy. To evaluate prediction accuracy, three of each model were trained on both domains. Unless otherwise specified, the Starcraft history distribution form distribution over [-10, - and the query distribution was set to fixed time offsets {1, 2, 4, 7)- Figure shows the validation loss for each model throughout training, and tables [ and show in detail the average error across each individual query the final trained networks predict for the test datasets. Our results show that TPC is significantly more accurate than the baseline algorithms,
set to be uni-
Table 1: Weather Nowcasting. Mean per-query prediction loss and 95% confidence interval for each target attribute across the test dataset. Normalized loss is calculated as described in section
Model
Rel. Humidity
Air Temp.
Wind Speed
Air Pressure
Normalized Loss
SeFT
22.97 457.36
1.65 43.46
0.87 +1.85
5.40 111.6
0.0299 +000003
DeepSets
25.07 #57.78
2.36 +4.11
0.99 #2.15
15.88 4:00.99
0.0383 #0.0004
Minkowski
260.72 4401.69
62.20 194.15
7.75 12232
9668.67 (.6..22226 16670.14
4.2803 10.0202
TPC (Ours)
9.75 433.67
1.43 42.41
0.54 41.26
3.75 14.89
0.0179 10.0002

---

Under review as a conference paper at ICLR 2021
TPC
SefT
Mink
Fixed1
Fixed2
Fixed2
-10
0.1
Uniferm
Uniform
0.0
-10
Ta
0.2
Tall
Fireed
Ficest2
Unitenys
Fail
Ficeed
Firest2
Unifarm
Fail
Fised1
Fieed2
Uniform
(a) Input Distribution Visualization
(b) Off Distribution Performance
Figure 3: Input Distribution Comparison. (aa shows the probability of selecting each timestep
as input during training for each distribution type. 3(b) depicts the relative performance of networks trained each input distribution when evaluated across all four. Note that Y-axes scaled by row.
pecially on the Starcraft II unit state prediction problem. In all cases, the Minkowski network was unable to outperform either of the set function-basee -based models, and in the weather nowcasting domain it consistently failed to find good solution, as indicated by the loss orders of magnitude higher than the set function approaches. We believe this failure due to the difficulty of selecting suitably sized kernel and voxelization resolution for spatio-temporal problem the scale of an entire state. We were unable to increase the size of the kernel without driving the network's 's parameter count prohibitively high, and we were unable to decrease the resolution of voxelization without starting to 'lose' a significant number of weather stations which would be occupying the same cell. This result suggests that applying 'true' point cloud convolution that directly exploits sample positions is preferable for these domains, as opposed to discretizing or voxelizing the samples' locations so that a traditional fixed-size filter convolution such as Minkowski networks can be applied.
Impact of Train and Test Distributions. distribution of input samples or query points. Since the TPC architecture is completely decoupled from the distribution of the imput samples, we can accomplish this comparison by simply defining several distribution types, training a model with each type of input distribution on the Starcraft II domain, and comparing the results after evaluating each trained model across each of the input distribution types selected for evaluation.
We investigate the robustness of TPC to change in the
We selected four input distributions for evaluation: Two 'fixed' distributions that always return the same set of time offsets, the uniform distribution over the range [-10, 0]. and half of a normal dis- tribution over the range 10, O]. Figure visualizes visualizess the difference between these distributions, and presents a bar chart plotting the average loss when each model evaluated on each distribution type. In all cases, the query distribution SeFT trained on fixed distributions perform poorly when evaluated on any distribution it was not trained on, while the Minkowski network suffers much less of penalty despite worse absolute per- formance. Alternatively, the networks trained on the uniform and normal distributions suffer much less degradation when switching different input distributions. The only case with noticeable
kept constant and fixed. The results show that TPC and
Table 2: Starcraft IL. Mean per-query prediction loss and 95% confidence interval of each algorithm for each target attribute on the test set.
Model
Pusition
Health
Shield
Orientation
Alive
Total Error
SeFT
0.139 4151
0.018
=0.0664
0.005 20.0177
1.260 1215
0.175 10.4472
1,600 10.0016
DeepSets
0.127 20.4158
0.014
=0.0393
0.004
10.0156
1.218 +1
0.158 40.429
1.524 10.0017
Minkowski
0.666 20.8572
0.131 20.1820
0.021 20.0532
1.865 10.5880
0.464 40.5166
3.141 10.0013
TPC (Ours)
0.083 =0.3502
0.006
=0.0187
0.002
=0.0084
1.017 +1.3501
0.102 40.3569
1.213 10.0018

---

Under review as a conference paper at ICLR 2021
Temporal PointConv
Fixed1
10'
Fixed2
10*
10c
Uniferm 0.1
Uniform10
10
0.2
Tail
10a
10c
0.0
10
0.0
2.5
5.0
7.5
10.0
12.5
15.0
Query Distribution Visualization
(b) Prediction Loss by Time Offset
Figure 4: Query Distribution Comparison. ((a shows the probability of selecting each timestep
query during training. H(b)showe loss queries by timestep up to 15 for networks trained on each query distribution.
performance drop for networks trained on the normal distribution and evaluated on the uniform distribution, which is unsurprising since the normal distribution is biased toward f = 0.
We perform similar experiment to evaluate the behavior of TPC when trained on different query distributions. Figure visualizes visualzees the query distributions selected for training alongside plot of the average loss for each query by their offset from the reference time (e.g. models trained on fixed distributions only consistently perform well on the exact query points they were trained on, with the model trained on Fixedl distribution's prediction error rising sharply as the distance from its small cluster of expected query points increases. In contrast, the model trained on the variable distributions saw a relatively small increase in prediction error, even for query points that are outside of the range of query points it was trained on. This suggests that the ability train the TemporalPointConv architecture on randomized input and query distributions. is key to enabling it generalize well across timesteps and behave reasonably in off-distribution scenarios.
0). before,
Application to Anomaly Detection. We now consider the utility of our TPC model for anomaly detection, where the goal is detect which samples in temporal point cloud are anomalous. We fo- cus on the weather dataset, where anomalies correspond to broken sensors. We introduce anomalies the of test station samples by randomly selecting 33% of the stations. For these, randomly increase or decrease the value of one station property by factor of 25%. The models are then tasked with predicting each of the test samples' properties given the preceding hour of weather data. Their prediction еrror on each individual sample is then used as anomaly score for detection purposes.
As expected based on prior prediction results, TPC significantly outperforms SeFT owing its su- perior nowcasting accuracy with an area under receiver-operator curve (AUROC) of 0.927 compared to SeFT's 0.836. The Minkowski network struggles to perform above chance level. See appendix A for the complete ROC curves.
CONCLUSION
In this work, we proposed a novel extension to the set function PointConv that enables to be composed with standard deep learning layers to reason about irregularly sampled spatio-temporal processes and calculate predictions for arbitrary domain-specific queries. We show that Temporal- PointConv's ability to directly consume each sample's 's positional and feature data without down- sampling or discretization enables it to significantly outperform state of the art sparse convolution algorithms across two complex, meaningfully different domains. Similarly, TemporalPoint equivalence to standard convolution enables it to more efficiently reason about relative spatial and temporal relationships than other set functions which are not endowed with these useful properties. These promising results and TemporalPointCom effectively applied to a wide range problems with an irregular structure that prevents most other deep learning approaches from functioning efficiently.
flexible parameterization suggest that it can be
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing 2016.
systems, pp.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3075-3084, 2019.
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
pp.
Benjamin Graham and Laurens van der Maaten. Submanifold sparse convolutional networks. In IEEE Conference Computer Vision and Pattern Recognition, 2018.
Hermosilla, T. Ritschel, P-P Vazquez, A. Vinacua, and T. Ropinski. Monte carlo convolution for learning on non-uniformly sampled point clouds. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2018), 37(6), 2018.
Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for time series. In International Conference Machine Learning. 2020.
Shuang Li, Shuai Xiao, Shixiang Zhu, Nan Du, Yao Xie, and Song. Learning temporal point processes via reinforcement learning. In Advances in neural information processing systems, pp- 10781-10791, 2018a.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. International Conference on Learning Representations, 2018b.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 652-660, 20.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet+++: Deep hierarchical feature learning point sets in metric space. In Advances in neural information processing systems, pp. 5099-5108, 2017b.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularlyssampler time series. In Advances in Neural Information Processing Systems, pp- 5330, 2019.
Satya Narayan Shukla and Benjamin Marlin. Interpolation-preeiitioo networks for irregularly sam- pled time series. In International Conference on Learning Representations, 2018.
Martin Simonovsky and Nikos Komodakis. Dynamic edge-conditioned filters in convolutional neu- ral networks on graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3693-3703 2017.
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming- Hsuan Yang, and Jan Kautz. Splatnet: Sparse lattice networks for point cloud processing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2530-2539, 2018.
Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6411-6420, 2019,
Oriol Vinyals, Timo Ewalds. Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani. Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv: 1708.07708.0488.... 2017.

---

Under review as a conference paper at ICLR 2021
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions on Graphics (TOG), 2019.
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 9621-9630, 2019.
SHI Xingjian, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In Ad- vances neural information processing systems, pp. 802-810, 2015.
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spiderenn: Deep learning on point sets with parameterized convolutional filters. In Proceedings of the European Conference Computer Vision (ECCV), pp. 87-102, 2018.
Huaxiu Yao, Xianfeng Tang, Hua Wei, Guanjie Zheng, and Zhenhui Li. Revisiting spatial-tempone similarity: A deep learning framework for traffic prediction. Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 33, pp. 5668 1866-5675,- 2019.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. International Joint Conference on Artificial Intelli- gence, 2018.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in neural information processing systems, 3391- 3401, 2017a.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poezos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in neural information processing systems, pp. 3391- 3401, 2017b.
Qiang Zhang. Aldo Lipani, Omer Kirap, and Emine Yilmaz. Self-attentive hawkes processes. arXiv preprint arXiv: 1907.0007700 0007007511 2019.
Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. process. arXiv preprint 2000:
Transformer hawkes
10

---

Under review as a conference paper at ICLR 2021
A
ANOMALY DETECTION ROC CURVES
AUC=0.9273
AUC=0.8364
AUC=0.500
1.0
1.0
0.3
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
(a) TemporalPointConv
(h) SeFT
(c) Minkowski
Figure 5: ROC curves of each model's prediction error thresholding anomaly detection performance.
B HYPERPARAMETER SETTINGS
Table 3: Hyperparameter settings used to instantiate each type of model on each type of domain.
Domain
Starcraf
Weather
Model
TPC
SeFT
DeepSets
Miniowesi
SeFT
DeepSets
Minkowski
PiiiCaa: Wiight Network Halden
16,
32, 32
NinCCaw: Camid
32
32
PoiniComy Final MLP Bidken
64,
64, 64
DeepSen: DerpSets Hidden
197. 197, 192
192, 197, 192
128. 128, 178 Yos
192, 192, 192.
DerpSet: Sclf Attention
Yoa
No.
No
Mink: Votel Resolution
0,05
6000 km
Mink: Kemee Size
21,21, 17
21,21,
Latent Neighbothood Feature Sizes
16,
16. 32,
32,
16,
16, 30,
16, 12,32,33 33
16. 32.
32,
TemporiPNint Encoder Hidlen
32,64,
32,
Max Neighbors
Latent Size
64
64
64
Decoder
64, 64,64,66
64,
(3e-4.
36-51
1e:6)
(1e-3, le-7)
Optinizer
ADAM
ADAM
ADAM
ADAM
ADAUM
ADAM
1.M
14%L
104M
C JOINT SPACE-TIME NEIGHBORHOODS
1.4 x 10
IPC-Combined-1 TPC-Combined 02
1.3 x 10
Combined
1.2 10
TPC
Loss
1.1 10"
10a
x -
0.2
0.4
0.6
0.8
1.0
Epoch
Figure Combined distance function experiment results.
Though TemporalPointConv decomposes spatio-temporal processes into separate 'space' and 'time' neighborhoods, this not strictly necessary. Space and time could be combined into one single vector space, allowing for a single PointConv layer jointly consider samples' spatial and temporal distances to determining their local neighborhood.
We investigate this possibility by training TemporalPointCom networks to do exactly that. This requires specifying space-time distance function which define follows: Dat VDz xD

---

Under review as a conference paper at ICLR 2021
where Dj and Dr are spatial and temporal distance functions, respectively. then represents the tradeoff factor that dictates whether distant spatial samples should be favored over temporally distant samples when constructing neighborhood.
Specifically, we test three values for x for these 'combined' PointConv models: 0.2. 1, and 5. The results in show that all of the networks with combined spatial-temporal neighborhood func- tions were outperformed by our approach which considers spatial and temporal relationships sepa- rately but sequentially. Additionally, this combined distance function depends hyperparamter which is likely domain-specific and nontrivial to find a good value for. These results validate our decision treat spatial and temporal distances separately.
12


Reviews:
[{"review_id": "AbvYeT7Um-G", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a new spatial-temporal point cloud processing technique, which extends the prior work of PointConv for spatial point processing to the temporal domain. Through experiments on two datasets, this paper shows improved performance over a few baselines.\n\nPaper Strengths:\n1. The direction of spatial-temporal point cloud processing is indeed underexplored so I appreciate the efforts put into this direction in the paper. \n2. The ability to answer arbitrary prediction queries could be useful in different application domains\n\nPaper Weaknesses:\n1. The technical contribution is limited. Essentially, the proposed method is just to apply the existing PointConv operator in both spatial and temporal dimensions. Also, regarding the literature of spatial-temporal point cloud processing, it would be nice if the paper can add some discussion/comparison with PointRNN [A1], which I think is the closest work to this paper though to my knowledge it is not published yet (has been on arXiv for more than a year).\n2. The proposed method seems to be problematic in modeling temporal dynamics. Specifically, the PointConv operator in Eq. 1 is to aggregate features within a neighborhood (weighted summation of the features), which does not consider the order of the points in the neighborhood, i.e., permutation-invariant. In the spatial point processing case, this is fine in some applications where we do not consider the direction but only distances of neighborhood points. However, in the temporal domain (i.e., Eq. 3), I feel using PointConv does not make much sense as the order of time should matter in general. Essentially, Eq.3 tells us that, if the centroid point is at frame t, then its corresponding neighbor point at frame t-H has the same distance measure as its corresponding neighbor point at frame t+H, which means that the direction of time is not considered. I feel that the formulation needs to inform the network of the time flow, i.e., the order of time. Also, in Eq 2, the neighborhood also considers some points within a small time window but discards the order of the time again. As a result, I do not think the spatial-temporal point processing proposed in this paper is properly modeling temporal dynamics. Even simple GNN/PointNet + RNN [Li et al 2018b, A2] network structure considers the order of time in the RNN.\n3. A big problem I feel about the proposed method is that it relies on a strong assumption about the point correspondences in time, which largely limits the practical impact of this paper. In this paper, the proposed method is shown to reason for high-level entities (e.g., represent an object or a Starcraft unit as a single point), which is fine as the correspondences can be obtained by object tracking in practice. However, how about the scenario where each point does not represent a high-level object but represent low-level observations, e.g., 3D point clouds obtained by LiDAR or stereo reconstruction? In such cases, it is extremely difficult to obtain the point correspondences across time as there could be no correspondences at all in the real-world. Then the proposed method might have problems applied to such data which is widely used in many applications such as point cloud-based object detection, SLAM, point cloud-based classification/segmentation/prediction. In contrast, Minkowski Networks [Choy 2019] that this paper is compared to and also the PointRNN [A1] and SPF [A2] can be applied to these more challenging scenarios. \n4. I am a bit concerned about the dataset used in the experiments as the data in the Starcraft II dataset seems to be generated randomly, which is hard to be reproduced and compared by the follow-up work. Would it be possible that evaluating the proposed method on the test set of a few public benchmarks (e.g., traffic prediction) that others can easily compare with even without re-implemented the proposed method? \n5. The re-implementation of the Minkowski Networks is somewhat concerning. The paper claims that it is hard to find a suitable kernel size and vocalization resolution, and will lead to a prohibitively high number of network parameters if increasing the kernel size. This leads me to wonder how large scale the data is, e.g., how many points and how many frames the method needs to process. If the scale of the data is not too big, there should not be a problem for the Minkowski Networks I guess. In the original paper of the Minkowski Networks, they evaluated their method on a sequence of large-scale LiDAR point cloud data (usually with 50k number of points per frame), which seems fine and obtains strong performance. It would be nice if a more detailed explanation of why the Minkowski Networks cannot be properly tuned.\n\nJustification:\n\nMy decision is made mainly because I feel the proposed method seems to have some flaws and limitations. Also, I am not fully convinced by the experimental results due to the concern about data and re-implementation, and the technical contribution is limited too. However, I would be happy to change my mind if there are any misunderstandings. \n\nReferences\n\n[A1] H. Fan and Y. Yang. PointRNN: Point Recurrent Neural Network for Moving Point Cloud Processing. arXiv 2019\n\n[A2] Weng et al. Inverting the Forecasting Pipeline with SPF2: Sequential Pointcloud Forecasting for Sequential Pose Forecasting. CoRL 2020\n\nPost-rebuttal Review\n\nAs there is no response submitted by the authors, I would like to stick to my original rating to reject this paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4g026068nTA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of modeling spatial-temporal point clouds which are sampled at irregular space and time points. It proposes the Temporal PointConv model which is an extension of the PointConv model (Wu et al., 2019). In particular, PointConv computes a convolution by aggregating the features of nearby points of a point p as the new feature of p.  Temporal PointConv extends this by aggregating the features of points near p in both space and time in a two-step process: first weighting the aggregation by the space distance and then weighting the aggregation by the temporal distance. \n\nExperiments on two datasets show that Temporal PointConv outperforms the baseline models in prediction accuracy. \n\nPros:\n1. The paper studies an interesting topic. Spatial-temporal point cloud modelling has many applications including weather forecasting as shown in the paper.  \n\n2. The experimental results are good. The proposed model achieves substantial improvement in terms of the prediction accuracy. The experiments done with a gaming setting (Starcraft II) is interesting. \n\n3. The paper is well written and easy to follow. \n\nCons:\n1. The paper proposed a simple and effective model. However, a concern is on its technical contribution. As discussed above, the proposed Temporal PointConv model is a direct extension of the PointConv model. The technical contribution is too little to justify a publication in a top-tier conference. \n \nAdditional comments:\nThe Abstract claims that the proposed model achieved improved efficiency compared to state-of-the-art baselines, but there is no experimental result reported on model efficiency. \n\nGrammar: \"do not have recorded properties we can use to determine property loss.\" => \"do not have recorded properties which we can use to determine property loss.\"\n\nTypo: \"tables 1 and 2\" => \"Tables 1 and 2\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "2nAEp5bbgbV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper spatio-temporal point convolution are proposed, which can be used for sequences of sparse and unordered data.  For this purpose, a spatial convolution and a temporal convolution are applied separately, which are then combined in a further step.The presented method was evaluated with two different data sets, based on Starcraft II and on data from weather stations.\n\nThe method is conclusive and the effectiveness is shown by interesting evaluations. Nevertheless, in my eyes there are still some open questions:\n* Why not simply use 4D convolution like for example in the paper Meteornet: Deep learning on dynamic 3d point cloud sequences? In general, the motivation of the method and its components is relatively brief.  For example, an ablation study, where the importance of the time component is shown, would be advantageous.\n* The data sets consist of sequences with corresponding point data. What about scan data, without any correspondence? It would be interesting to see a test with semantic segmentation (with lidar data) or action estimation (from depth images) to see more general applications of the TPC layer.\n* It is not clear to me why graph convolution networks are not taken into account, in my eyes graph networks would be a good choice for the given data. Recent publications show the abilities of GCNs in similar application areas (e.g. Learning to Simulate Complex Physics with Graph Networks).\n\nThe paper itself is written relatively clear and understandable, a few little things I noticed: \n* Section 4.3: ... to transform each latent query representation into a final prediction~~**s**~~ ...\n* 5.2 Paragraph 3: I think the reference to Figure 6 was confused with the reference to Figure 3.\n* Figure 3 and Figure 4: I would leave the scale of the Y-axis of the distribution visualization the same, otherwise it could be misleading.\n\nIn summary, I find the method interesting, but in my opinion, it still needs some improvements and more elaboration regarding evaluations. Therefore I tend to a reject.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1dxdzOf3io", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an extension of PointConv for spatial-temporal point cloud modeling. The model can be used for prediction or forecasting and is evaluated on Starcraft II and weather nowcasting.\n\n- The TemporalPointConv follows PointConv and the current work extends this by appending time. I think the paper is heavily based on PointConv which makes the overall novelty limited.\n\n- The temporal convolution either does not track points or relies on entity labels for tracking. As in real and most point-cloud scenarios point labels are not available, the temporal convolution restricts the applications of the proposed method. \n\n- Starcraft II and weather nowcasting are not common applications of point cloud and I thus feel the evaluation is not convincing enough. It would be more appealing to apply the method to other applications such as autonomous driving cars.\n\n- Making weather nowcasting via point clouds has been studied [a]. Discussion and comparison with related works are missing. \n[a] CloudLSTM: A Recurrent Neural Model for Spatiotemporal Point-cloud Stream Forecasting\n\n- I think the idea of appending time dimension can also be applied to PointNet++, KPConv, or other operations. Why do we choose PointConv? Is there any problem if the proposed method is based on other operations?\n\n=========Post Rebuttal==========\nBecause no response is provided, I maintain my original rating.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new spatial-temporal point cloud convolution. However, many reviewers suggest the paper can be improved with better baselines and motivations.
Decision: Reject