CloudLSTM: A Recurrent Neural Model for Spatiotemporal Point-cloud Stream Forecasting
Authors: Chaoyun Zhang
Marco Fiore
Iain Murray
Paul Patras
Venue:  (2020)
Keywords: spatio-temporal forecasting
point cloud stream forecasting
recurrent neural network

Abstract: This paper introduces CloudLSTM, a new branch of recurrent neural models tailored to forecasting over data streams generated by geospatial point-cloud sources. We design a Dynamic Point-cloud Convolution (D-Conv) operator as the core component of CloudLSTMs, which performs convolution directly over point-clouds and extracts local spatial features from sets of neighboring points that surround different elements of the input. This operator maintains the permutation invariance of sequence-to-sequence learning frameworks, while representing neighboring correlations at each time step -- an important aspect in spatiotemporal predictive learning. The D-Conv operator resolves the grid-structural data requirements of existing spatiotemporal forecasting models and can be easily plugged into traditional LSTM architectures with sequence-to-sequence learning and attention mechanisms.
    We apply our proposed architecture to two representative, practical use cases that involve point-cloud streams, i.e. mobile service traffic forecasting and air quality indicator forecasting. Our results, obtained with real-world datasets collected in diverse scenarios for each use case, show that CloudLSTM delivers accurate long-term predictions, outperforming a variety of neural network models.

CLOUDLSTM: A RECURRENT NEURAL MODEL FOR SPATIOTEMPORAL POINT-CLOUD STREAM FORECASTING
ABSTRACT
This paper introduces CloudLSTM, branch of recurrent neural models tai- lored to forecasting over data streams generated by geospatial point-cloud sources. We design Dynamic Point-cloud Convolution (DConv) operator the core com- ponent of CloudI_STMss which performs convolution directly over point-clouds and extracts local spatial features from of neighboring points that surround different elements of the imput. This operator maintains the permutation invariance learning frameworks, while representing neighboring correlations at each time step an important aspect in spatiotemporal predictive learning. The DConv operator resolves the grid-structural data requirements of existing spatiotemporal forecasting models and can be easily plugged into tra- ditional LSTM architectures with sequence-to-sqqueme mechanisms. We apply our proposed architecture to two representative, practical use cases that involve point-cloud streams, i.c., mobile service traffic forecasting and air quality indicator forecasting. Our results, obtained with real-world datasets collected in diverse scenarios for each use case, show that CloudLS" L delivers accurate long-term predictions, outperforming variety of neural network models.
learning and attention
INTRODUCTION
Point-cloud stream forecasting aims at predicting the future values and/or locations of data streams generated by geospatial point-cloud S, given sequences of historical observations (Shi & Yeung 2018). Example data sources include mobile network antennas that serve the traffic generated by ubiquitous mobile services at city scale (Zhang et al. 2019b), sensors that monitor the air quality of a target region (Cheng et al. 2018), or moving crowds that produce individual trajectories. Unlike traditional spatiotemporal forecasting 2013) or video frame prediction (Wang et al., 2018), point-cloud stream forecasting needs to operate on geometrically scattered sets of points, which are irregular and unordered, and encapsulate complex spatial correlations. While vanilla Long Short-term Memories (LSTMs) have modest abilities to exploit spatial features (Shi al. 2013), convolution- based recurrent neural network (RNN) models, such ConvLSSTV (Shi et al. (015) and PredRNN+ (Wang et al.) (2018), are limited to modeling grid-structural data, and are therefore inappropriate for handling scattered point-clouds.
grid-structura data, like precipitation nowcasting (Shi et al.,
Forecasting over grids
Figure Different approaches to geospa- tial data stream forecasting: predicting over input data streams that are inherently grid-structured, e.g., video frames using ConvLSTMs (top); mapping point- cloud input to grid, e.g., mobile net- work traffic collected at different anten- in city, enable forecasting using existing neural network structures (mid- dle); forecasting directly over point-cloud data streams using historical information (as above. but without pre- pre-processing). as proposed in this paper (bottom).
Forecasting over point clouds by grid transforaation
BM+1
++1
t+
Forecasting point clouds

---

Leveraging the location information embedded in such irregular data sources, to learn important spatiotemporal features, is in fact challenging. Existing approaches that tackle the point-cloud stream forecasting problem can be categorized into two classes, both of which bear significant shortcomings: (i) methods that transform point-clouds into data structures amenable to processing with mature solutions, e-g., grids exemplified in Fig T (Zhang et al. 2019ap; and (ii) models that ignore the exact locations of each data source and inherent spatial correlations (Liang et al. 2018). The transformations required by the former not only add data preprocessing overhead, but also introduce spatial displacements that distort relevant correlations among points (Zhang et al. (019a). On the other hand, the latter are largely should be revisited over time, to suit series prediction tasks (Shi et al. (017)) In essence, overlooking dynamic spatial correlations will lead to modest forecasting performance.
while recent literature suggests spatial correlations
Contributions. In this paper, we introduce Convolutional Point-cloud LSTMs (CloudL.STH) (TMs), a new branch of recurrent neural network models tailored to geospatial point-cloud stream forecasting. The CloudLSTM builds upon a Dynamic Point-cloud Convolution (DConv) operator, which takes raw point-cloud streams (both data time series and spatial coordinates) as input, and performs dynamic convolution over these, to learn spatiotemporal features over time, irrespective of the topology and permutations of the point-cloud. This eliminates the need for the data preprocessing described above, and avoids the distortions thereby introduced. The proposed CloudL.STM. takes into account the locations of each data source and performs dynamic positioning at each time step, to conduct a deformable convolution operation over point-clouds (Dai et al. [017). This allows revising the spatiotemporal correlations and the configuration of the data points over time, and guarantees the oecation-invarian property is met at different steps. Importantly, the DConv operator flexible, as it can be easily plugged into existing neural network models for different purposes, such as RNNs, LSTMs, sequence-to to-eeuuecee (Seq2seq) learning (Sutskever et al. 2014). and attention mechanisms (Loong et al. 2015).
We evaluate our proposed architectures on four benchmark datasets, covering two spatiotemporal point-cloud stream forecasting applications: (i) antenna-level forecasting of data traffic generated by mobile services (Zhang & Patras 2018 Bega et al. 2019), leveraging metropolitan-scale mobile traffic measurements collected in two European cities for 38 popular mobile apps; and (ii) forecasting six air quality indicators on two city clusters in China (Zheng et al. 2015). These tasks represent important use cases of geospatial point-cloud stream forecasting. We combine our CloudLSTM with Seq2seq learning and an attention mechanism, then undertake a comprehensive evaluation on all datasets. The results demonstrate that our architecture can deliver precise long-term point-cloud stream forecasting in different settings, outperforming 12 different baseline neural network models in terms of four performance metrics, without any data preprocessing requirements. To the best of knowledge, the proposed CloudLSTM is the first dedicated neural architecture for spatiotemporal forecasting that operates directly on point-cloud streams.
2 RELATED WORK
Spatiotemporal Forecasting, Convolution- based RNN architectures have been widely employed for spatiotemporal forecasting, as they simultancously capture spatial and temporal dynamics of the input. Shi et al., incorporate convolution into LSTMs, building a ConvLSTM for precipitation nowcasting (Shi et af. 2015). This approach exploits spatial information, which in turn leads to higher prediction accuracy. The ConvLSTM is improved by constructing a subnetwork predict state- to-state connections, thereby guaranteeing location-variance and flexibility of the model (hi et al. 2017). PredRNN (Wang al. 2017) and PredRNN+ (Wang et al., 2018) evolve the LSM by constructing spatiotemporal cells and adding gradient highway units. These improve long-term forecasting performance and mitigate the gradient vanishing problem in recurrent architectures. Although these solution work well for spatiotemporal forecasting, they can not be applied directly to point-cloud streams, as they require ppint-cloud-to-gria preprocessing (Zhang et al. (019a)).
Feature Learning on Point-clouds. Deep neural networks for feature learning point-cloud data are advancing rapidly. PointNet performs feature learning and maintains input permutation invariance (Qi et al.] 2017a). PointNet++ upgrades this structure by hierarchically partitioning point-clouds and performing feature extraction on local regions (Qi et al. (0076)). VoxeINet employs voxel feature encoding to limit inter-poine interactions within voxel (Zhou & Tuzel 2018). This effectively projects cloud- cloud-points onto sub-grids, which enables feature learning. Li et al., generalize the convolution operation on point-clouds and employ X' x-transforrations learn the weights and
2

---

permutations for the features (Li al. 2018 Through this, the proposed PointCNN leverages spatial-loca correlations of point clouds, irrespective of the order of the input. Notably, although these architectures can learn spatial features of point-clouds, they are designed to work with static data, thus have limited ability to discover temporal dependencies.
3 CONVOLUTIONAL POINT-CLOUD LSTM
Next, we describe in detail the concept and properties forecasting over point cloud-streams. We then introduce the DCony operator, which is at the core of our proposed CloudLSTM architecture. Finally, we present CloudLSTM and its variants, and explain how to combine CloudLSTM with Seq2seq learning and attention mechanisms, to achieve precise forecasting point-cloud streams.
3.1 FORECASTING OVER POINT-CLOUD STREAMS
We formally define a point-cloud containing set of N points, as S {21, pN). Each point Pv. E contains two sets of features, i.e., Pn. = {Vo, Sn where V,, = (...... ; are value features (e.g., mobile traffic measurements, air quality indexes, etc.) of Pn. and Sn fc,..... cck are its L-dimensional coordinates. At each time step t, we may obtain U different channels of S by conducting different measurements denoted by S = (S ... ) ) S/¹ E RUXNNx Here, different U resemble the RGB channels in images. We can then formulate the J-step point-cloud stream forecasting problem, given M observations, as:
argmax P(S ST.11" 1t
S44J ST
1+1
D-J
Note that, in some cases, each point's s coordinates may be unchanged, since the data sources are deployed at fixed locations. An ideal point-cloud stream forecasting model should embrace five key
properties, similar to other point-cloud applications and spatiotemporal forecasting problems (Qi
jt al. 2017aa Shi et al.] 2017):
(i) Order invariance: A point cloud is usually arranged without specific order. Permutations of the
input points should not affect the output of the forecasting (Qi et al.) 2017a).
(ii) Information intactness: The output of the model should have exactly the same number of points as the input, without losing any information, i.c., Nout = Nin-
(iii) Interaction among points: Points in S are not isolated, thus the model should be able to capture local dependencies among neighboring points and allow interactions (Qi et al. 2017a).
(iv) Robustness to transformations: The model should be robust to petrrelaaioo-peseevvii transfor- mation operations on point-clouds, c.g., scaling and shifting (Qi et al. 2017a).
(v) Location variance: The spatial correlations among points may change over time. Such dynamic correlations should be revised and learnable during training (Shi et al. (017).
In what follows, we introduce the Dynamic Point Cloud Convolution (DConv) operator as the core module of the CloudL STM, and explain how DConv satisfies the aforementioned properties.
3.2 DYNAMIC CONVOLUTION OVER POINT CLOUD
The Dynamic Point Cloud Convolution operator (DConv) generalizes the ordinary convolution on grids. Instead of computing the weighted summation over small receptive field for each anchor point, DConv does so on point-clouds, while inheriting desirable properties of the ordinary convolution operation. The vanilla convolution takes Un channels of 2D tensors as input, and outputs Uout channels of 2D tensors of smaller size (if without padding). Similarly, the DConv takes Un channels of point-cloud S, and outputs Uout channels of a point-cloud, but with the same number of elements as the input, to ensure the information intactness property (īi) discussed previously. For simplicity, we denote the ith channel of the input set as Si and the jth channel of the output as S Both S and Sout 3D tensors, of shape (N, (H L.), Um) and (N, (H T.), Uout) respectively.
We also define O as a subset of points in Si., which includes the K. nearest points with respect ....,, ). where pt: is the k-thh nearest point
to pu in the Euclidean space, i.e., OK = .p...... ........pp ph. to Pr, in the set St Note that pn itself is included in @K as an anchor point, i.c., Pr. p Recall that each Pn. E S contains H value features and L coordinate features, i.c., P.. = {Vus S, ). where en = [v/, ... o and Sn. (ch, .4). Similar the vanilla convolution operator, for each p..
3

---

Figure 2: Illustration of the DConv op- erator, with single input channel and K = neighbors. For every E S DCony weights its K neighboring set 2 = {ph ,P to produce values and coordinate features for pl, E S Here, each w is set of weights with index h (i.e., k-th nearest neighbor) in Eq. 2 shared across different p.
Ph
PulP
Pon
in Sin the DConv sums the element-wise product over all features and points in O to obtain the values and coordinates of point pl, in S. Note that we assume the value features are related to their positions at the previous layer/state, to better exploit the dynamic spatial correlations. Therefore, we aggregate coordinate features c(p when computing the value features oh The mathematical
expression of the DConv is thus:
un
Σ
Σ
Σ NCH
A. Wi "(Ph)
Σ TEL
Wij
(H-D)M& &
c(phi)) + bjj
iEUm pi EQ!
Σ
Σ NCH
A.l' o(p wes
+
Σ
wis
(H+1)
c(ptil) +b3)
iEU PRERK
let
Sut = (ph,... .P'N)
(2)
-(f''.... of (..)..)... (woNs .(.....
(Ch.....
okkk)
In the above, we define learnable weights W 5D tensors with shape (Uin: KC, (1 L), (H L), Uout). The weights are shared across different anchor points in the input map. Each element wi k W a scalar weight for the i-th input channel, j-th output channel, k-th nearest neighbor of each point corresponding to the m-th value and coordinate features for each input point, and m' value and coordinate features for output points. Similar to the convolution operator, we define by bias for the j-th output map. In the above, 1t and h are the - value features of the input/output point set. Likewise, 1 and l' are the 1 coordinate features of the input/output. (-)) the sigmoid function, which limits the range of predicted coordinates to (0, 1), to avoid outliers. Before feeding them to the model, the coordinates of raw point-clouds are normalized to (0, 1) by < = (s - (mmmm))/// (Smax - Smin), on each dimension. This improves the transformation robustness of the operator.
The nearest points can vary for each channel at each location, because the channels in the point- cloud dataset may represent different types of measurements. For example, channels in the mobile traffic dataset are related to the traffic consumption of different mobile apps, while those in the air quality dataset are different air quality indicators (SO2, CO, etc.). The spatial correlations will vary between different measurements (channels), due to human mobility. For instance, more people may use Facebook at a social event, but YouTube traffic may be less significant in this case. This will be reflected by the data consumption of each app. The same applies to air quality indicators affected by vehicle movement and factory working times. We want these spatial correlations to be learnable, so we do not fix the K. nearest neighbors across channels, but encourage each channel to find the best neighbor set. This is also a contribution of the CloudLSTM, which helps improve the forecasting performance.
We provide a graphical illustration of DConv Fig. 2 For each point Pr., the DConv operator weights its IC nearest neighbors across all features, to produce the values and coordinates in the next layer. Since the permutation of the input neither affects the neighboring information nor the ranking of their distances for any QK. DConv is a symmetric function whose output does not depend on the input order. This means that the property (i) discussed in Sec. 3.1 is satisfied. Further, DConv is performed every point in set S and produces exactly the same number of features and points for its output; property (ii) is therefore naturally fulfilled. In addition, operating over a neighboring point set, irrespective of its layout, allows to capture local dependencies and improve the robustness
4

---

to global transformations (e.g., shifting and scaling). The normalization over the coordinate features further improves the robustness to those transformations, as shown by the proof in Appendix [C] This enables to meet the desired properties (iti) and (iv). More importantly, DConv learns the layout and topology of the cloud-point for the next layer, which changes the neighboring set 2 for each point output S This enables the "location-varian dynamic positioning tailored to each channel and time step. This is essential in spatiotemporal forecasting neural models, spatial correlations change over time (Shi et al.] 2017). DConv can be efficiently implemented using simple 2D convolution, by reshaping the input map and weight tensor, which can be parallelized easily in existing deep learning frameworks. We detail this in Appendix A and provide complexity analysis of the DConv operator in Appendix
(property (v)), allowing the model to perform
Relations with PointCNN (Li et al., 2018) and Deformable Convolution Dai et al., 2017). The DConv operator builds upon the PointCNN (Li et al.) (2018) and deformable convolution neural network (DefCNN) on grids (Dai et al.] 2017), but introduces several variations tailored to point- cloud structural data. PoinCCNN employs the X-transformation over point clouds, to learn the weight and permutation on local point set using multilayer perceptrons (MLPs), which introduces extra complexity. This operator guarantees the order invariance property, but leads to information loss, since it performs aggregation over points. In our DConv operator, the permutation is maintained by aligning the weight of the ranking of distances between point pa and O Since the distance ranking is unrelated to the order of the inputs, the order invariance is ensured in a parameter-free manner without extra complexity and loss of information.
Further, the DConv operator can be viewed as the DeICNN (Dai et al. 2017) over point-clouds, with the differences that (i) DefCNN deforms weighted filters, while DConv deforms the input maps; and (ii) DefCNN employs bilinear interpolation over input maps with a set of continuous offsets, while DConv instead selects K. neighboring points for its operations. Both DefCNN and DCony have transformation modeling flexibility, allowing adaptive receptive fields on convolution.
3.3 THE CLOUDLSTM ARCHITECTURE
The DConv operator can be plugged straightforwardly into LSTMs, learn both spatial and temporal correlations over point-clouds. We formulate the Convolutional Point-cloud LSTM (CloudLSTM as:
aWar SH + Wha + H b,).
ft = oW., C St + Way + bb)),
C. fi O C--1 + i O tanh(Wsc S + Whe Hi-1 + be),
(3)
0 = (W... + S + Who + H bo).
H. 0, @ tan)
Similar to ConvLSTM (Shi et al., (015)), i, fcr and o,, are input, forget, and output gates respectively. C, denotes the memory cell and H. is the hidden states. Note that i, ft, ot, C.. and H, are all point cloud representations. W and represent learnable weight and bias tensors. In Eq. 3 "O' denotes the element- wise product, 'O' is the DConv operator formalized in Eq. 2 and " a simplified DConv that removes the sigmoid function in Eq. The latter only operates over the gates computation, as the sigmoid functions are already involved in outer calculations (first, second, and fourth expressions in Eq. 3). We show the structure of a basic CloudLSTM cell in the left part of Fig.
We combine CloudLSTM with Seq2seq learning (Sutskever et al. 2014) and the soft attention mechanism (Luong et al. 2015), to perform forecasting, given that these neural models have been proven be effective in spatiotemporal modelling on grid-structural data (e.g., (Shi et al. 2015 Zhang et al., 2018)). We show the overall Seq2seq CloudLSTM in the right part of Fig. 3 The architecture incorporates an encoder and a decoder, which are different stacks of CloudLSTMs. The encoder encodes the historical information into tensor, while the decoder decodes the tensor into predictions. The states of the encoder and decoder are connected using the soft attention mechanism via a context vector (Luong et al.) 2015). Before feeding the point-cloud to the model and generating the final forecasting, the data is processed by Point Cloud Convolutional (CloudCNN) layers, which perform the DConv operations. Their function is similar to the word embedding layer in natural language processing tasks (Mikolov et al., 2013). which helps translate the raw point-cloud into tensors and vice versa. In this study, we employ two-stack mucoder-decoder architecture, and configure 36 channels for each CloudLSTM cell, as we found that further increasing the number of stacks and channels does not improve the performance significantly.
5

---

Oulpul gale
Embedding
Clar
Encoder
Alation
Ni+s
1
Fil alll Stn
Inpul gale
Mr
16
H4-
Dousluth
Co CoutEsT
Forget gate
Decoder
Embeddiig
Si
Sy-Me1
Figure 3: The inner structure of the CloudLSTM cell (left) and the overall Seq2seq CloudLSTM architecture (right). We denote by (.)" and (-)5 the value and coordinate features of each input, while these features are unified for gates.
Beyond CloudLSTM, we also explore plugging the DConv into vanilla RNN and Convolutional GRU, which leads to new Convolutional Point-cloud RNN (CloudRNN) and Convolutional Point-cloud GRU (CloudGRU), as formulated in the Appendix E The CloudRNN and CloudGRU share a similar Seq2seq architecture with CloudLSTM, except that they do not employ the attention mechanism. We compare their performance in the following section.
4 EXPERIMENTS
To evaluate the performance of our architectures, we employ measurement datasets of traffic generated by 38 mobile services and recorded at individual network antennas, and of 6 air quality indicators collected at monitoring stations. We use the proposed CloudLSTM to forecast future mobile service demands and air quality indicators in the target regions. We provide a comprehensive comparison with 12 baseline deep learning models, over four performance metrics. All models considered in this study are implemented using the open-source Python libraries TensorFlow (Abadi et al. 2016) and TensorLaye (Dong et al. 2017). We train all architectures with a computing cluster with two NVIDIA Tesla K40M GPUs We optimize all models by minimizing the mean square error (MSE) between predictions and ground truth, using the Adam optimizer (Kingma & Ba 2015).
Next, we first introduce the datasets employed in this study, then discuss the baseline models used for comparison. Finally, we report on the experimental results obtained.
4.1 DATASETS AND PREPROCESSING
We conduct experiments on two typical spatiotemporal point-cloud stream forecasting tasks over geospatial environments, with measurements collected in two different scenarios for each case. Note that, as the data sources have fixed locations these applications, the coordinate features will be omitted in the final output. However, in different use cases, such crowd mobility forecasting, the coordinate features would be necessarily included.
Mobile Traffic Forecasting. We experiment with large-scale multi-service datasets collected by major operator in two large European metropolitan areas with diverse topology and size during 85 consecutive days. The data consists of the volume of traffic generated by devices associated each of the 792 and 260 antennas in the two target cities, respectively. The antennas are non non-uniformly distributed over the urban regions, thus they can be viewed as 2D point clouds over space. At each antenna, the traffic volume is expressed in Megabytes and aggregated over 5-minute intervals, which leads to 24,482 traffic snapshots. These snapshots are gathered independently for each of 38 different mobile services, selected among the most popular apps for video streaming, gaming, messaging, cloud services, and social networking. Further details about the dataset can found Appendix G
Air Quality Forecasting, Air quality forecasting performance is investigated using public dataset (Zheng et al., 2015), which comprises six air quality indicators (i.e., PM2.5, PM10, NO2, CO, O and SO₂) collected by 437 air quality monitoring stations in China, over span of year. The monitoring stations are partitioned into two city clusters, based on their geographic locations, and measure data an hourly basis. The dataset includes 8,760 snapshots in total for each cluster. We conduct experiments on both clusters individually and fill missing data using linear interpolation. The reader is referred to Appendix Go details.
Before feeding to the models, the measurements associated to each mobile service and air quality indicator transformed into different input channels of the point-cloud S. All coordinate features

---

Table I: The mean mean=ssd std of MAE, RMSE, PSNR, and SSIM across all models considered, evaluated on two datasets collected in different cities for mobile traffic forecasting.
City
2
Model
MAE
RMSE
PSNK
SSIM
MAE
RMSE
PSNK
SSIM
MLP
0.79
9.9422.56
49.56=2.13
0.27=0.12
0.59
9.44=2..5
50.30=22228
0.33=00144
CNN
6.00+:0.22
11.02+22.09
48.93=1.60
0.25::0.12
5,30-00.51
10.05:22.06
49.97+1.877
0.32=00114
3D CNN
0.95
9.94.12.4
49.74.2.13
0.33110.14
5.21110.88
2.97
50.131 1.85
0.37.10.16
DefCNN
6.76:1:0.11
11.72:42.57
18.42:1.822
0.16:10.8
5.31400.11
9.99-12.13
19.84:11.87
0.32-1.114
PointCNN
0,95++0.53
10.10: 46
49.43=22.06
0.27 12
0.75=00.56
9.55=22.32
20.1742.16
0.35=0.115
CloudCNN
4.811-0.88
9.91:12..1
49,91422.11
0.29-10.11
0.68:10.52
9.39.12.22
50.31
0.36-0.14
LSTM
4,20-+:0.66
9.58:43.117
30.47=3.29
0.36++0.10
1.32+1..64
9.17=33.03
50.7943.26
0.42=0012
ComvLSTM
3.98+11.60
9.25
30.47
0.36+0.10
4.09:11.59
8.87=22.97
51.10=3.33
0.42-#0.12
PredRNN++
3.97::1160
9.29:13.12
30.43:3..30
0.36:10.100
1.07
8.87-12.977
51.09:3.34
0.42:10.12
PointI.STM
4.63.1...5
9.47.12.55
50.02
0.34.10.14 14
0.56.10.54
9,26.11.43
20.52.1
0.37.10.15
CloudRNN (K
9)
1.06
9.19
3,245
0.32:10.12
1.08
3.74:13.03
31.10:3.266
0.39::014
CloudGRU (KC 9)
1.79+1.59
8.90=3.111
30,73+3.29
0.39+ 10
1.90
8.47=2296
51,40=3.33
0.45=0.12
Cloudl. STM (X 3)
1.71
8.87.2..11
30.7611.30
0.39-10.11 10
3.86 1.51
8.42:1...4
31.45
0.46-1.11
CloudLSTTM (KC
3.72+11.3
8.1133.13
30.72=3.299
0.38-20.10
1.59
8.46+22.96
51.4343.33
0.45=0.12
CloudL.STM (KC = 9)
3.72++1.22
8.88:+3.11
30,75=3.229
0.39:
1.89:11.55
8.46::2.96
$1.41#3322
0.46:10.11
Attention
CloudLSTM (AC 9)
3.66
8.12
3..78.13.21
0.40:10.11
1.79
8.43:12.96
51.36
0.47:10.11
normalized to the 10 range. In addition, for the baseline models that require grid-structural input (i.c., CNN, 3D-CNN, ConvLSTM and PredRNN++), the point-clouds are transformed into grids (Zhang et al., 2019a) using the Hungarian algorithm (Kuhn 1955). The ratio of training plus validation, and test sets is 8:2,
4.2 BENCIMARKS AND PERFORMANCE METRICS
We compare the performance of our proposed CloudLSTM with a set of baseline models, as follows. PointCNN (Li et al.) 2018) performs convolution point-clouds and has been employed for point-cloud classification and segmentation. CloudCNN is an original benchmark we introduce, which stacks the proposed DConv operator over multiple layers for feature extraction from point- clouds. PointLSTM is another original benchmark, obtained by replacing the cells in ConvLSTM with the X'-Conv operator employed by PointCNN, which provides a fair term of comparison for other Seq2seq architectures. Beyond these models, we also compare the CloudLSTM with two of its variations, i.e., CloudRNN and CloudGRU, which were introduced in Sec. 3.3 Other baseline
models, including MLP (Goodfellow et al., 2016), CNN (Krizhevsky et al. 20122) 3D-CNN (Ti et al.
2013), LSTM (Hochreiter Schmidhuber 1997), ConvLSTM (Shi et al. 2015) PredRNN++ (Wang
et al. 2018). along with the detailed configuration of all models are discussed in Appendix
We quantify the accuracy of the proposed CloudL STM in terms of Mean Absolute Error (MAE) and Root Mcan Square Error (RMSE). Since the mobile traffic snapshots can be viewed as "urban images" (Liu et al. al. 2013). we also select Peak Signal-to-Noiss Ratio (PSNR) and Structural Similarity Index (SSIM) (Hore & Ziou| 2010) to quantify the fidelity of the forecasts and their similarity with the ground truth, as suggested by relevant recent work (Zhang et al. 2017). Details about the metrics discussed in Appendix
For the mobile traffic prediction task, we employ all neural networks to forecast city-scale mobile traffic consumption over time horizon of J 6 sampling steps, i.e., 30 minutes, given M 6 consecutive past measurements. For RNN-based models, i.e., LSTM, ConvLSTM, PredRNN++ CloudLSTM, CloudRNN, and CloudGRU, we then extend the number of prediction steps i.c., 3 hours, to evaluate their long-term performance. In the air quality forecasting use case, all models receive a half day of measurements, i.e., . 12, as input, and forecast indicators in the following 12 hours, i.e., J = 12. As for the previous use case, the number of prediction steps are then extended to = 72, or 3 days, for all RNN-based models.
36,
4.3 RESULT ON MOBILE TRAFFIC FORECASTING
We perform 6-step forecasting for 4,888 instances across the test set, and report in Table | the mean and standard deviation (std) of each metric. We also investigate the effect of a different number of neighboring points (i.e., KC 3, 6, 9), well as the influence of the attention mechanism. Observe that RNN-based architectures in general obtain superior performance, compared to CNN- based models and the MIP. In particular, our proposed CloudL and its CloudRNN, and
7

---

City 1
City 2
Average MAE
12
PIcORNK
CouaLSTM (C-
12
PredRWK-
CluSLSTN
PORILSTM
ShuuttS (e-
PariLSTM
CluuRRNN
Alleniinn CloudL51M
GoudRMK
Altendion ClousLSIM
5
10
15
20
25
30
10
15
20
25
30
35
Prediction step
Prediction step
Figure 4: MAE evolution wrt. prediction horizon achieved by RNN-based models on both cities for
mobile traffic forecasting.
CloudGRU variants outperform all other banchmark architectures, achieving lower MAE/RMSE and higher PSNR/SSIM on both urban scenarios. This suggests that the DConv operator learns features over geospatial point-clouds more effectively than vanilla convolution and PointCNN. Among approaches, CloudLSTM performs better than CloudGRU, which in turn outperforms CloudRNN.
Interestingly, the forecasting performance of the CloudLSTM seems fairly insensitive to the number of neighbors (IC;; it is therefore worth using a small K. in practice, to reduce model complexity. Further, observe that the attention mechanism improves the forecasting performance, as it helps capturing better dependencies between input sequences and vectors in decoders. This effect has also been confirmed by other NLP tasks. We provide a complete service-wise evaluation in Appendix H
Long-term Forecasting Performance. We extend the prediction horizon to up to J = 36 time steps (i.e.. 3 hours) for all RNN-based architectures, and show their MAE evolution with respect this horizon in Fig. 4 Note that the input length remains unchanged. i.c., 6 time steps. In city 1, observe that the MAE does not grow significantly with the prediction step for most models, as the curves flatten. This means that these models are reliable in terms of long-term forecasting. As for city 2, we note that low I. may lead to poorer long term performance for CloudLSTM though not significant before step 20, This provides guideline choosing K. for different forecast lengths required.
4.4 RESULTS ON AIR QUALITY FORECASTING
We employ all models to deliver 12-step air quality forecasting on six indicators, given 12 snapshots as input. Results are in Table [ Also in this use case, the proposed CloudLSTMs attain the best performance across all 4 metrics, outperforming state-of-the-art methods (ConvLSTM) by up to 12.2% and 8.8% terms of MAE and RMSE, respectively. Unlike in the mobile traffic forecasting results, a lower AC yields better prediction performance. though the difference appears subtle. Again, the CloudCNN always proves superior to the PointCNN, indicating that CloudCNNs are better feature extractors over point-clouds. Overall, these results demonstrate the effectiveness of the CloudLSTM models for modeling spatiotemporal point-cloud stream data, regardless the tasks to which they applied. Performance evaluations of long- term forecasting, i.e., up 72 future time steps, conducted on RNN-based models and results are presented in Appendix[[
Note that conduct our experiments using strict variable-controllinng methodology, i.e., only changing one factor while keep the remaining the same. Therefore, it is easy to study the effect of each factor. For example, taking look at the performance of LSTM. LSTVLSTM, PredRNN++. PointLSTM and CloudLSTM, which employ dense layers, and CNN, PointCNN and D-Conv as core operators but using LSTM as the RNN structure, is clear that the D-Conv contributes significantly to the performance improvements. Further, by comparing CloudRNN, CloudGRU and CloudL.TMM, appears that CloudRNN & CloudGRU < CloudLSTM Similarly, by comparing the CloudLSTM and Attention CloudL we see that the effects of the attention mechanism are not very significant. Therefore, we believe the core operator > RNN structure > attention, ranked by their contribution.
CONCLUSION
We introduce CloudLSTM, a dedicated neural model for spatiotemporal forecasting tailored to point- cloud data streams. The CloudLSTM builds upon the DConv operator, which performs convolution over plont-clouds to learn spatial features while maintaining permutation invariance. The DCony simultaneously predicts the values and coordinates of each point, thereby adapting to changing spatial correlations the data at each time step. DCony is flexible, as it can be easily combined with various RNN models (i.e., RNN, GRU, and LSTM), Seq2seq learning, and attention mechanisms.
8

---

Table 2: The mean mean-std of MAE, RMSE, PSNR, and SSIM across all models considered, evaluated on two datasets collected in different city clusters for air quality forecasting.
Cluster
Clsstr
Model
MAE
RMSE
PSNK
SSIM
MAE
RMSE
PSNK
SSIM
MLP
113.1=119 191.9
242.03=240.24
23.3447.5
0.13+0.10
40.34=22.16
50.3 27.27
24.75=4.02
0.10=0.10
CNN
37.6244..18
11.67=11.21
28.35+22388
0.13+00.5
18.5942.224
23.66+22.66
30.17+1114
0.34=0.04
3D CNN
37.09.17.63
48.01
10.36
28.36.12.17
0.32.10.88
19.84.12.20
25.30.12555
29.58
29.58.10.99
0.05
DefCNN
37.5148.34
47.69::1:44
28.40:12.11
0.131:0.05
19.46::22.5
25.58:4..80
29.55-1.07
0.26=0.05
PointCNN
39.60=47.63
11.65
27.63++2.02
0.19:00.4
19.25++2.38
24.60#+2.99
29.89=
0.17=0.03
CloudCNN
29.914:2.89
31.914201
LSTM
30,62++8.97
11833-1188
29.87+2.779
0.31
14,38:43.37
10-0-04422
32.16=2.03
0.41=0.77
ConvLSIM
22.91+:8.09
31.62=11.40
31.98+3.224
0.50++0.10
10.39:42.82
14.20++3.87
34.79=2.45
0.60=0.06
PredRNN++
25.14:18.48
34.38-11.77
31.34:1..13
0.37-10.08
11.43:422.1
15.68
33.94422211
0.50=0.05
Point STM
36.64.17.99
47.42.10.66
28.56.12.25
0.31.10.06
18.77.42.18
24.66.1...58
29.79.11.07
0.35
CloudRNN (R
33.09:18.23
42.16:11.388
2.65342.66
0.13:10.07
34.72:43.75
19.70:14.54
31.93:2.18
0.14=0.06
CloudGRU (KC
9)
22.12++8.02
30.65=11.25
32.22++.38
0.53+t0.09
9.58=2282
1341
35.26=2.57
0.68=0.07
CloudLSTM 3)
ClondI STM (K:
20.84:1...88
29.16:11.M8
32.64-13.40
0.57:10.10
2.12
12.95:1..72
28.59-1..22
0.69-0.07
21.31:7.52
10.71
32.48+3.29
0.55+00.10
9.38=2 2.85
13:20:22.79
35.42:22.00
0.68=0.07
CloudLSTM. (KC 9)
21.72:47.83
30.14=11.05
32.36:2..34
0.54::0110
9.73+2284
13,58:3.77
35.20:12.56
0.66=0.07
CloudLSTM (AC 21.72.17.78
Attention
10.95
32.38.13.29
0.56.10.10
9.38.12.69
13.41.13.78
35.26.12.57
0.07
We consider two application case studies, where we show that our proposed CloudL .STM ISMM state-of-the-ar performance on large-scale datasets collected in urban regions in Europe and China. We believe the CloudLSTM gives new perspective on point-cloud stream modelling, and can be easily extended to higher dimension point-clouds, without requiring changes to the model.
achieves
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, al. TensorFlow: A system for large-scale machine learning. In OSDI, volume 16. pp. 265-283, 2016.
Dario Bega, Marco Gramaglia, Marco Fiore, Albert Banchs, and Xavier Costa-Perez. DeepCog: Cognitive Network Management Sliced 5G Networks with Deep Learning. In Proc. IEEE INFOCOM, 2019.
Jon Louis Bentley, Multidimensional binary search trees used for associative searching. Communica- tions of the ACM, 18(9):509-517, 1975.
Weiyu Cheng, Yanyan Shen, Yanmin Zhu, and Linpeng Huang. A neural attention model for urban air quality inference: Learning the weights of monitoring stations. Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764-77777 773, 2017.
Hao Dong, Akara Supratak, Luo Mai, Fangde Liu, Axel Oehmichen, Simiao Yu, and Yike Guo. TensorLayer: A versatile library for efficient deep learning development. In Proc. ACM Multimedia, pp. 1201-1204, 2017.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
/ww.deee-earringgook. org
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Alain Hore and Djemel Ziou. Image quality metrics: PSNR vs. SSIM. In Proc. IEEE International Conference on Pattern Recognition (ICPR), pp. 1366623699,22 2010.
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3D convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 35(1):221-25 2013.

---

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffryy E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Harold W Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics (NRL), 2(1-2):83-97, 1955.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. PointCNN: Convolution on X-rransfomeed points. In Advances in Neural Information Processing Systems, pp. 820-830, 2018.
Yuxuan Liang, Songyu Ke, Junbo Zhang, Xiuwen Yi, and Yu Zheng. GeoMAN: Multi-level attention networks for geo-sensory time series prediction. In IJCAI, pp. 3428-3434, 2018.
Liang Liu, Wangyang Wei, Dong Zhao, and Huadong Ma. Urban resolution: New metric for
measuring the quality of urban sensing. IEEE Transactions on Mobile Computing, 14(12):2560- 2575, 2015.
Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421, 2015.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 31 11131999, 2013.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652 660, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. +ointNet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems, pp- 5099-5108, 2017b.
Xingjian Shi and Dit-Yan Yeung. Machine learning for spatiotemporal sequence forecasting: A survey. arXiv preprint arX1v:1808.0666. 2018
Xingjian Shi, Zhourong Chen, Hao Wang. Din-Yan Yeung. Wai-Kin Wong, and Wang-chun Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems, pp. 802-810, 2015.
Xingjian Shi, Zhihan Gao, Leonard Lausen, Hao Wang, Dit- Yan Yeung, Wai-kin Wong, and Wang- chun Woo. Deep learning for precipitation nowcasting: A benchmark and a new model. In Advances in Neural Information Processing Systems, pp. 5617-5627, 2017.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 3212, 2014.
Yunbo Wang, Mingsheng Long, Jianmin Wang. Zhifeng Gao, and S Yu Philip. PredRNN: Recur- rent neural networks for predictive learning using spatiotemporal Istms. In Advances in Neural Information Processing Systems, pp. 879-888, 2017.
Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and S Yu Philip. PredRNN++: Towards resolution of the deep-in-time dilemma in spatiotemporal predictive learning. In International Conference on Machine Learning, pp. 5110-5119, 2018
Chaoyun Zhang and Paul Patras. Long-term mobile traffic forecasting using deep spatio-temporal neural networks. In Proc. ACM MobiHoc, 2018
Chaoyun Zhang, Xi Ouyang, and Paul Patras. ZipNet-GAN: Inferring fine graingrined mobile traffic patterns via a generative adversarial neural network. In Proceedings of the 13th International Conference on emerging Networking EXperiments and Technologies, pp. 363-375. ACM, 2017.
10

---

Chaoyun Zhang, Marco Fiore, and Paul Patras. Multi- mobile traffic forecasting via convolu- tional long Short- Term memories. In 2019 IEEE International Symposium on Measurements & Networking (M&N) (IEEE M&N 2019), Jul 2019a.
Chaoyun Zhang, Paul Patras, and Hamed Haddadi. Deep learning mobile and wireless networking: A survey. IEEE Communications Surveys & Tutorials, 2019b.
Liang Zhang, Guangming Zhu, Lin Mei, Peiyi Shen, Syed Afaq Ali Shah, and Mohammed Ben- namoun. Attention in convolutional LSTM for gesture recognition. In Advances in Neural Information Processing Systems, pp. 1953-1962, 2018.
Yu Zheng, Xiuwen Yi, Ming Li, Ruiyuan Li, Zhangqing Shan, Eric Chang, and Tianrui Li. Forecasting fine-grained air quality based on big data. In Proceedings of the 21th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 2267-2276. ACM, 2015.
Yin Zhou and Oncel Tuzel. VoxeINet: End-to-end learning for point cloud based 3D object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4490- 4499, 2018.

---

APPENDIX
A DCONV IMPLEMENTATION
The DConv can be efficiently implemented using a standard 2D convolution operator, by data shape transformation. We assume batch size of for simplicity. Recall that the input and output of DConv, S and Sout, are 3D tensors with shape (N, (H + L), Um) and (N, (H + L), Uout), respectively. Note that for each Pr, in S we find the set of top K. nearest neighbors Q Combining these, we transform the input into 4D tensor S with shape (N, (H + L), Un. To perform DConv over Si split the operator into the following steps:
Algorithm 1 Efficient algorithm for DCony implementation using the 2D convolution operator
1: Inputs:
S with shape K, (H + L.), U.m)
2: Initialise:
The weight tensor W.
3: Reshape the input map S from shape (N, (H + L), Um) to shape (N, (H L) x Um)
4: Reshape the weight tensor W from shape (Uin. KC, (H LD, L), ) to shape (1, IC, Unn X
(H + L), Unut (H L))
5: Perform 2D convolution Sout Conv(S S W) with step 1 without padding. Sout becomes
3D tensor with shape 1, Uot (H + L))
6: Reshape the output map Sout to (N, (H + L), )out)
7: Apply the sigmoid function (-) to the coordinates feature in Sout
This enables to translate the DConv into a standard convolution operation, which is highly optimized by existing deep learning frameworks.
B DCONV COMPLEXITY ANALYSIS
We study the complexity of DConv by separating the operation into two steps: (i) finding the
neighboring set O for each point Pa E S, and (ii) performing the weighting computation in Eq. 2 We discuss the complexity of each step separately. For simplicity and without loss of generality, we assume the number of input and output channels are both 1. For step (i), the complexity of finding
K. nearest neighbors for one point is close to O(K- L log N) if using KD trees (Bentley 1975).
For step (ii), it is easy to see from Eq. 2 that the complexity of computing one feature of the output P', is O( + L) 1.). Since each point has (H + L) features and the output point set S has N points, the overall complexity of step (ūi) becomes O KC (H + 2) This is equivalent the complexity of vanilla convolution operator, where both the input and output have (H - L) channels, and the input map and kernel have N and K elements, respectively. This implies that, compared the convolution operator whose inputs, outputs, and filters have the same size, DConv introduces extra complexity by searching the nearest neighbors for each point O(K L log N). Such complexity does not increase much even with higher dimensional point clouds.
C PROOF OF TRANSFORMATION INVARIANCE
We show that the normalization of the coordinates features enables transformation invariance with shifting and scaling. The shifting and scaling of point can be represented as:
Aç + B,
'L < log(n) is required to guarantee efficiency. In real life the dimensions of a point cloud dataset are normally 2 or 3, and we usually have significantly more than points in the dataset. Hence, this condition should hold for most applications.
12

---

where A and B are a positive scaling coefficient and respectively offset. By normalizing the coordinates, we have:
Smn
Smax
Smin
(As + B) (Asmin + B)
(Asmax + B3) (Asmin + B)
(5)
Als - Smin)
A Smax Smin)
Smin
Smax
Smin
This implies that, by using normalization, the model is invariant to shifting and scaling transforma- tions.
D SOFT ATTENTION MECHANISM
We combine our proposed CloudLSTM with the attention mechanism introduced in (Luong et al. 2015). We denote the j-th and i-th states of the encoder and decoder as Hi en and Hi de. The context
Tensor for state i at the encoder can be represented as:
eld
Σ Rig H JEM
G
ΣjEM Cij
where et, is score function, which can be selected among many alternatives. In this paper, we W.
choose ei,j = v" tanh(W, * Hfel). Here [: is the concatenation operator and is the convolution function. Both Wa and V. are learnable weights. The H and context tensor are concatenated into tensor for the following operations.
E MODELS CONFIGURATION
We compared proposal against set of baselines models. MLP (Goodfellow et al. 2016). CNN (Krizhevsky et al., 2012). and 3D-CNN eJ et al. 2013) are frequently used as benchmarks in mobile traffic forecasting (Zhang & Patras 2018 Bega et al. 2019). DefCNN learns the shape of the convolutional filters and has similarities with the DCony operator proposed in this study (Dai al. 2017). LSTM is an advanced RNN frequently employed for time series forecasting (Hocrreter & Schmidhuber 1997). While ConvLSTM (Shi et al. 2015) can be viewed as baseline model for spatiotemporal predictive learning, the PredRNN+ the state-of-the-ard architecture for spatiotemporal forecasting on grid-structural data and achieves the best performance in many applications (Wang et al. 2018). The CloudRNN and CloudGRU have can be formulated as:
CloudGRU:
a(W., ② ST Wh. ② H + bz), Tt = a aWWrr Sy Whr H by), H tanh(r, (r, WNs H We's S.) H. (1 2)) @ H + 2 O H
CloudRNN:
(Wah S. + Way Vt-1 1 bb), (7) = (W h4 b,)
(8)
The CloudRNN and CloudGRU share similar Seq2seq architecture with CloudL.STM, except that they do not employ the attention mechanism.
We show in Table the detailed configuration along with the number of parameters for each model considered in this study. Note that we used 2 layers (same as our Clou)LSTMM)) for COmVLSTM. PredRNN+ and PointLSTM, since found that increasing the number of layers did not improve their performance. 3 x filters are commonly used in image applications, where they have been proven effective. This yields receptive field of 9 (3 3), which is equivalent to K. in CloudLSTMs. Thus this supports a fair comparison. In addition, the PredRNN+ follows a slightly different structure than that of other Seq2seq models, as specified in the original paper.
13

---

Table 3: The configuration of all models considered in this study.
Model
Configuration
MLP
Five hidden layers, 500 hidden units for each layer
Eleven 2D convolutional layers, each applies 108 channels and 3 filters, with batch normalization and ReLU functions. Eleven 3D convolutional layers, each applies 108 channels and 3 3 x 3 filters, with batch normalization and ReLU functions.
CNN
3D-CNN
Eleven 2D convolutional layers, each applies 108 channels and 3 x 3 filters, with batch normalization and ReLU functions. Offsets are
DefCNN
predicted by separate convolutional layers
PointCNN
Eight X-Convv layers, with K. D. P. C=[9, 1, 1,, 36]
CloudCNN
Eight DConv layers, with 36 channels and K. = 9
L.TTM
2-stack Seq2seq LSTTM, with 500 hidden units
ConVLSTM
2-stack Seq2seq ConvL.STM, with channels and 3 x 3 filters
PredRNN+
2-stack Seq2seq PredRNN++, with 36 channels and 3 x 3 filters
PointLSTM
2-stack Seq2seq PointLSTM, with K. D. P. [=[9, 1, -1. 36]
CloudRNN
2-stack Seq2seq CloudRNN, with 36 channels and K. 9
CloudGRU
2-stack Seq2scq CloudGRU, with 36 channels and Æ. 9
CloudISTTM (K. = 3)
2-stack Seq2seq CloudLSTTM, with 36 channels and K. = 3
CloudLSTM (AC 6)
2-stack Seq2seq CloudLSTTM, with 36 channels and
6
CloudLSTM (AC 9)
2-stack Seq2seq CloudLSTM, with 36 channels and K. 9
2-stack Seq2seq CloudLSTM, with 36 channels, K. = 9 and soft attention mechanism
Attention CloudLSTM
F
LOss FUNCTION AND PERFORMANCE METRICS
We optimize all architectures using the MSE loss function:
MSE(() I
ΣΣ -1)00-000)) NEH
Here Th is the mobile traffic volume forecast for the /-th service, and respectively the forecast value of the h-th air quality indicator, at antenna/monitoring station n, time while v is the corresponding ground truth. We employ MAE, RMSE, PSNR and SSIM to evaluate the performance of our models. These are defined as:
MAE()) = IN [
Σ nEN nEN hEH
Σ
(1) (1))
(10)
RMSE(1) V IN H
ΣΣ
(1 (1)11
(11)
REN REH
PSNR(1) = log Umaa (1) -10
Σ Σ nEW NEA
(0%(()) - (1)P.
(12)
NI H
SSIM(t) (2 (t) (t) + a) (2 (ov (h)) + (t)² (t) -ci ci) (VAR(C)(())VAR(()C) + +22
(13)
where (t) and Umax(() (1) are the average and maximum traffic recorded for all services/quality indicators, at all antennas/moontorinng stations and time instants of the test set. VAR(-) and c-v(.) denote the variance and covariance, respectively. Coefficients e₁ and 02 are employed to stabilize the
fraction in the presence of weak denominators. Following standard practice, we set c = Lk2 and 22 T222. where I. 2 is the dynamic range of float type data, and kn 0.1, l2 0.3.
14

---

City
City 2
Figure 5: The anonymized locations of the antenna set in both cities.
City
Traffre fraction (8)
201
Streaming
Meesaging Gaming Lancela
Cloud
2002
SAN
7
6%%
City
Traffre fraction (N)
101
Streaming
Others
Good
OUPS
Social kets
Miccelancous
120
Messactin
Gaming
17%
ASP
000
100
000
Figure Fraction of the total traffic consumed by each mobile service (left) and each service category (right) in the considered set.
G DATASET STATISTICS
G.1 MOBILE TRAFFIC DATASET
G.1.1 DATA COLLECTION
The measurement data is collected via traditional flow-level deep packet inspection at the packet gateway (P-GW). Proprietary traffic classifiers are used associate flows specific services. Due to data protection and confidentiality constraints, we do not disclose the name of the operator, the target metropolitan regions, or the detailed operation of the classifiers. For similar reasons, we cannot name the exact mobile services studied. We show the anonymized locations of the antennas sets in both cities in Fig. [
As final remark on data collection, we stress that all measurements were carried out under the supervision the competent national privacy agency and in compliance with applicable regulations. In addition, the dataset employ for our study only provides mobile service traffic information accumulated at the antenna level, and does not contain personal information about individual sub- scribers. This implies that the dataset is fully anonymized and its use for our purposes does not raise privacy concerns. Due to a confidentiality agreement with the mobile traffic data owner, the raw data cannot be made public.
15

---

Stations City Cluster
B) Stations in City Cluster
Figure 7: Geographic distribution of 437 air quality monitoring stations in both city clusters. Figures adapted from the readme.pd pdf file included with the dataset.
G.1.2 SERVICE USAGE OVERVIEW
As already mentioned, the set of services S considered in amalysis comprises 38 different services. An overview of the fraction of the total traffic consumed by each service and each category both cities throughout the duration of the measurement campaign is in Fig. The left plot confirms the power law previously observed in the demands generated by individual mobile services. Also, streaming the dominant type of traffic, with five services ranking among the top ten. This is confirmed in the right plot, where streaming accounts for almost half of the total traffic consumption. Web, cloud, social media, and chat services also consume large fractions of the total mobile traffic, between 8% and 17%, whereas gaming only accounts for 0.5% of the demand.
G.2 AIR QUALITY DATASET
The air quality dataset comprises air quality information from 43 cities in China, collected by the Urban Computing Team at Microsoft Research. total, there are 2,891,30 1,393 air quality records from 437 air quality monitoring stations, gathered over a period of one year. The stations into two clusters, based on their geographic locations, as shown in Fig. Cluster A has 274 stations, while Cluster includes 163. Note that missing data exists in the records and gaps have been filled through linear interpolation. The dataset is available at https::////// www www.osoottt.c..oooooooooooo.oome com/ en.us//eeeaara//poojje///aaaapap airr/
partitioned
H SERVICE-WISE EVALUATION
We dive deeper into the performance of the proposed Attention forecasting accuracy for each individual mobile service, averaged over 36 steps. To this end, we present the MAE evaluation on service basis (left) and category basis (right) in Fig. Observe that the attention CloudLSTMs obtain similar performance over both cities at the service and category level. Jointly analyzing with Fig. 6 we see that services with higher traffic volume on average (e.g., streaming and cloud) also yield higher prediction errors. This is because their traffic evolution exhibits more frequent fluctuations, which introduces higher uncertainty, making the traffic series more difficult to predict.
by evaluating the
16

---

City
Streaming
20
Gaming
Web
Messaging
MAE
15
Social Nets
Others
10
11
5.00
5.
10 34
10:16
04
06.06
16.33
00.17
5.5
1.06
10.23
1p.61
1.
100
36
36
7
10
0
00.08
10:39
5.67
00.18
2.18
3:23
9.77
3.17
18
1.39
City 2
10
Streaming
20
Messaging Social
Miscellaneous
MAE
10
981
5.82
128
041
0.25
06
106
00.2
5575
(0.26
jp.73
97
16,88
1.22
M.77
218
1
10,90
7-24
144
58
0
133
2
39
00.
0.26
2.08
Service
Category
Figure 8: Mobile service-level MAE evaluation both cities for the Attention SIousLSTMSs
averaged over 36 prediction steps.
City
City
LETV
CouRtN
(6-
LST Cenut PrectiM
CoucRine
CoupLST cc-St
Average MAE
60
Conu
CTV (c--1
CoodETM
Asentio GoudLSN
Covnt Me
:50
25
40
30-
20:
10
5
10 15 20
25 30
55 Prediction step
35
40
45
50
60
65
70
5
10
55 60 70 15 20 25 35 40
45 50 Prediction step
Figure 9: MAE evolution wrt. prediction horizon achieved by RNN-based models on both city clusters for the air quality forecasting.
LONG-TERM AIR QUALITY FORECASTING
We show the MAE for long-term forecasting (72 steps) of air quality on both city clusters Fig. Generally, the error grows with time for all models, as expected. Turning attention the CloudLSTWV with different IC, though the performance of different settings appears similar at the beginning, larger K can significantly improve the robustness of the CloudLSTM, the MAE grows much slower with time when K- 9. This is consistent with the conclusion made in the mobile traffic forecasting task.
J RESULTS VISUALIZATION
J.1 HIDDEN FEATURE VISUALIZATION
We complete the evaluation of the mobile traffic forecasting task by visualizing the hidden features of the CloudLSTMM LSMM, which provide insights into the knowledge learned by the model. In Fig. TO we show an example of the scatter distributions of the hidden state He CloudLSTM and Attention CloudLSTM at both stacks, along with the first input snapshots. The first 6 columns show the H; for encoders, while the rest are for decoders. The input data snapshots are samples selected from City 2 (260 antennas/points). Recall that each H, has I value features and 2 coordinate features for each point, therefore each scatter subplot Fig. shows the value features (volume represented by different colors) and coordinate features (different locations), averaged over all channels. Observe that most subplots, points with higher values (warm colors) tend to aggregate into clusters and have higher densities. These clusters exhibit gradual changes from higher to lower values, leading to comet-shape assemblages. This implies that points with high values also come with tighter spatial correlations, thus Cloudl STMs learn aggregate them. This pattern becomes more obvious in
17

---

Encoder
Cloudl STM
Decoder
Stack
Starkk 2
Attention CloudLSTM
Stack
Stack 2
1-4
1-3
1-2
1-1
1+1
1+2
1+3
1+4
145
Figure 10: The scatter distributions of the value and coordinate features of the hidden state in H, for CloudLSTM and Attention CloudLSTM. Values and coordinates are averaged over all channels.
LSTH
Attontion
Step
Step
Step
Step
Step
Step
Step
Step
Step 10
Step
Step 12
Figure 11: NO; forecasting examples in City Cluster A generated by all RNN-basee based models.
stack 2, features extracted at higher level, exhibiting more direct spatial correlations with respect to the output.
J.2 PREDICTION RESULTS VISUALIZATION
Lastly, in Fig. 11 and 12 and we show a of NO, forecasting examples both city cluster A and considered for air quality prediction, generated by all RNN-hased models, offering performance comparison from purely visual perspective. Point-clouds are converted into heat maps using 2D linear interpolation. The better prediction offered by (Attention) CloudLSTMs is apparent, proposed architectures capture trends in the point-cloud streams and deliver high long-term visual fidelity, whereas the performance of other architectures degrade rapidly in time.
18

---

COIL
ROW
GRU
S
Step
Step 3
Step
Step
Step
Step
Step
Step
Stop 10 Stop
Step 12
Figure 12: NO₂ forecasting examples in Cluster B generated by the RNN- based models.
K ROBUSTNESS TO OUTLIERS
The DConv uses Sigmoid functions to regularize the coordinate features of each point, such that those points which are far from others will move closer each other and be more involved in the computation. Further, by stacking multiple DConv via dedicated structure (LSTM), the CloudLSTM has much stronger representability and therefore allows refine the positions of each input point at each time step and each stack. Eventually, each point can learn to move to the position where is best to be and therefore, our model continues to work well while forecasting with outlier points.
For demonstration, we use the density- based spatial clustering of applications with noise (DBSCAN) to find those outliers (red points) in both clusters the air quality dataset, as shown in Fig. 13 For each city cluster, the DBSCAN algorithm finds 16 outlier points, which are relatively isolated and far from the point-cloud center. We recompute the MAE and RMSE performance especially for these outlier points, shown in Table Observe that CloudLSTM still obtains the lowest prediction error as compared to the other models considered. Taking closer look at the CNN-based models, the CloudCNN, which employs the DConv operator, obtains the best forecasting performance relative CNN, 3D-CNN, DefCNN and PointCNN.
We dive deeper into the robustness of CloudLSTM to outliers by conducting experiments under more controlled scenarios. To this end, we randomly selected 50 weather stations in each city cluster and construct toy dataset. Among these weather stations, we randomly pick 10 outliers, and move their positions away from the center by d = {0. 0.5, 1, 5} on both x and y axes. The direction of movement depends on the quadrant of each outierr Note that the original position of each weather station is normalized to [0, 1]. so d 5 means the point is moved at a distance times the maximum range of original position. The positions of the remaining weather stations remain unchanged. We show the positions of each weather stations after moving by different d for both city clusters in Fig. 14 and [5
We retrain the CloudLSTM and PointLSTM under the same settings, and show the MAE and RMSE performance of each Table. 5) Observe that the proposed CloudLSTM performs almost equally inliers and outliers, irrespective of the distance to outliers. Importantly, CloudLSTM achieves significantly better performance over its counterpart PointLSTM. This further demonstrates that our model is robust outliers, whose locations appear "lone".
well when forecasting
19

---

Cluster A
Cluster B
Outliers Inliers
Outliers Inliers
y-axti
x-axis
x-axis
Figure 13: Clustering results in the two city clusters using DBSCAN,
Table 4: The std MAE and RMSE across models considered, evaluated on outlier points in different city clusters for air quality forecasting.
Cluster
Cluster
Model
MAE
RMSE
MAE
RMSE
MLP
109.851 195,49
136.36:1240.7
45.59-20.82
56.72:125.31
CNN
33.04
42.63=14.449
21.67+2.88
26.53+33.65
3D-CNN
32.36=t10.48
41.49-11.773
24.91
30.82+4.91
DefCNN
10.18
43.0+:14118
25.7241..3
33.06:14.26
PointCNN
34.71
44.56:114.65
24.56.42.88
30.79+13.49
CloudCNN
31.79=10.72
40.52+13.63
15.54+3.54
19.26:14.91
LSTM
27.15
35.85+14.31
15.62.44.13
19.72
ConvLSTM
21.5719.79
18.94414.16
11.1841..6
14.5
PredRNN+
24.34+9.82
32.78+13.95
12.53+3.55
16.43+4..98
PointLSTM
32.2+11.011
41.39+15.25
21.28
26.6345.38
CloudRNN (KC 9)
32.3418.57
40.17+12.6
15.01
18.60+5.23
CloudGRU (KC 9)
20.5219.67
17.49+13.86
9.59=3.01
12.6444.32
CloudLSTM (K 3)
18.7919.44
25.59=13.76
9.04=3.14
11.99:4..4
CloudLSTM (KC 6)
20.348.99
27.23=13.12
9.54=3.25
12.55+14.54
CloudL (KC 9)
20.09
27.24
9.77+33.5
12.79++4.68
Attention
19.7949.48
26.63+13.66
9.5+3.14
12.49-14.48
CloudLSTM (KC 9)
0.5
1.34
5.84
0.8
0.A
0.25
0.35
-0.65
0.2
-0.11
-0.4
Hiles Outlers
-0.40
-5.98
-1.98
0.2
0.6
-0.3
5.70
1.66
1.42
-0.35
1.36
3.92
-2.73
5.92
X-axis
K-axis
X-axis
Figure 14: Positions of weather stations after moving outliers towards the edge with different distances d in city cluster A.
20

---

d 0
0.5
d=
0.8
1.09
1.39
3.7
0.6
0.4
0.32
0.22
-0.8
-0.07
-2.37
-2.7
0.0
-0.46
-2.98
-4.96
-0.10
155
-2.60
X-
X-axis
X-axis
x-axis
Figure 15: Positions of weather stations after moving outliers towards the edge with different distances d in city cluster B.
Table 5: The mtd MAE and RMSE across CloudLSTM (K 9) and PointLSTM, evaluated on inliers and outliers of two sub-datasets collected in different city clusters for air quality forecasting.
Claster A
Claster B
Model
MAE
RMSE
MAE
RMSE
Intiers
Ouihers
Inbers
Outhers
Inliers
Outhers
Inliers
Outhers
22.16:77.28
20.3929.61
29.53=9.39
28.3528.76
9.78=3.63
9.06
13.10:3.76
13.28444.56
CloudLSTM
22.75:17:29
20.518.66
29.1629.44
28.80:1.225
10.07:1:.50
9.35
13.4943..3
13.59:14.13
22.411..30
19.58
29.04.19.58
28.41 18.36
10.03.15.440
9.21
13.33.13.26
13.56.14.99
21.9847.311
19.549.78
28.61-9.52
28.24.8.3
9.34=5.52
9.64
12.5143.63
12.69444.06
20.724200001
38.06:118:35
58.08:34.11
58.30+31.69
20.08=10:: 10.39
18.72=8.33
28.38:14.007
23.12213:22
PoinLLSTM
0.5
38.45=17.117
19.144166.64
55.32:+:11.2
57.48:1:0.06
19.97-99.53
19.49=6.98
28.16:113.69
27.88:10884
38.08111755
54.851311.4
54.11.31.48
19.29.19.65
18.40=7.35
27.38.113886
26.84:1.2.1
36.41:66.70
39.1341 16.93
52.55:130.82
56.8111:9982
17.49:17.56
20.34-9.34
25.05:111.03
29.35:113.67
L ADDITIONAL EVALUATION WITH SIMPLE BASELINES
We further compare our proposal with simple baselines, which also perform forecasting based on k- nearest neighbors each target point. To this end, we construct MLPs and LSTMs with the structures specified in 3 but with different input form. Specifically, for each point, the models perform prediction using only the K. nearest neighbors' data, with K. from {1. 3, 6, 9, 25, 50, 100). We show their performance along with that of our CloudLSTM on the air quality dataset in Table Observe that our CloudLSTM significantly outperforms MLPs and LSTMs, which conduct forecasting only relying k-nearest neighbors. The number of neighbors K. affects the receptive field of each model. small K. means the model only relies on limited local spatial dependencies, while global spatial correlations between points are neglected. In contrast, a large I. enables looking around larger location spaces, while this might lead to overfitting. The results in the table suggest that the K. does not affect the performance of each baseline significantly. Meanwhile our proposed CloudL STM, which extracts local spatial dependencies through DConv kernels and merges global spatial dependency via stacks time steps and layers, is superior to these simple baselines.
Table 6: The meanistd of MAE, RMSE, PSNR, and SSIM across k-nearest neighbors baseline models and our CloudLSTM, evaluated on two datascts collected in different city clusters for air quality forecasting.
Cluster A
Cluster B
Model
MAR
RMSE
PSNR
SSIVI
MAR
RMSE
PSNR
SSIVI
MLP = 1)
31.28
38.72.19.68
29.95.12.28
0.4810.11
14.33.12.06
17.77.12.78
32.20:11.33
0.61:0.06
29.12.17.33
38.36-19.42
30.1742.266
0.4610.10
15.35-42.12
19.20
31.16:1.111
0.59-0.06
MLP (K 6)
29.53-17.24
38.35=10.01
30.06+22.52
0.45+0.12
16.73-42.22
20.99+2.331
30.66+11011
0.58=0.05
MLP (KC 9)
29.73=1.94
38.62+111.12
30.15+2.77
0.40-0.10
18.734+2.40
21.98-+2.64
30.23+11.09
0.58=0.06
MLP(K (K 25)
31.59:17.63
40.34:1:1.39
29.82:12.54
0.30:10.09
18.93:1247
22.98:1..25
30.37:1.31
0.22-0.04
MLP (K 50)
30.528.02
39.02:110.72
30.074:2.60
0.29::0.12
14.6242.44
18.34:3.37
32.39:1155
0.54=0.77
100)
38.99:111.33
30.11:2.78
14.27413.32
18.6511.113
32.32-1195
0.39:00.5
LSTM (AC = 1))
24.31.17.52
10.65
31.67.1..02
0.53-10.10
11.471291
15.50
34.51
0.67.1..77
(K=
24.33-17.51
32.46-1 10.57
31.5443.113
0.5340.110
11.46:2294
15.45-3.96
0.65-0008
LSTM (K -6)
24.66:27.77
32.97=10.61
31.49+2.79
0.53+20.10
11.52+22.89
15.46+4.13
34,50+2.47
0.64=0.77
LSTM (IC = -9)
24.89-1..8
33.07-110.64
31.55:2..98
0.52-40.10
11.59:13.00
15.77:14.09
34.40+2.52
0.62-0.07
LSTM (KC
75.96
33.9.110.99
31.33.1300
0.50.10.10
11.08:3.06
15.7411.12
34.47
0.60-0.06
LSTM (K 50)
25.68:17887
33.98:110.1
31,27422.97
0.88:0.110
11.72433.26
13.81+428
34.4842.67
0.59=0.06
100)
25.57-18.73
34.53
31.1713.17
0.4410..9
13.77
33.97-12.22
0.55-10.05
CIoRLLSTMI
20.84=7.88
29.16::11.06
32,64=3.00
0,57+0.10
9.12=2775
12,95:3772
35.5942.62
0.69-000
21

---

Table 7: The mean-fstd of MAE and RMSE across all models considered without/with seasonal
information, evaluated on a subset of antennas in City for mobile traffic forecasting.
Minutes Window
30 Minutes
Days Window
Model
MAE
RMSE
MAE
RMSE
MLP
4.86=t0551
10.30=2252
0.93
11.30=2.2
CNN
6.10
11.12:2.04
5.98+0.60
10.52+2.11
3D-CNN
5.01++0.51
9.89-12.54
4.8240.54
9.49422.66
DefCNN
6.79-+0.89
11.92-+2.47
6.40++0.83
11.55++2.33
PointCNN
5.01
10.22+2.40
4.88
9.86122.11
CloudCNN
4.79:10.33
9.9412.75
4.63+0.50
9.5742.66
LSTM
4.2440.64
9.67+3.23
4.04+0.67
9.28+3303
ConvLSTM
4.10+1.61
9.28+3.11
3.82+1.554
8.87+3.00
PredRNN++
3.94
9.31=3.10
3.61+1.55
8.95+2292
PointLSTM
4.63:10.41
9.4412.46
4.44:10.41
9.0042.32
CloudRNN (K 9)
4.14#1.67
9.18=3.13
3.99#1.62
8.88+2.93
CloudGRU (KC 9)
3.77+1158
8.95+
3.42+11.3
8.53++2.88
CloudLSTM (K. 3)
3.6911.61
8.8323.09
3.38
8.40+2.86
CloudLS STM (K. 6)
3.68+1.60
8.87+3.10
3.39+1.54
8.43++2.76
CloudLSTMM (K. 9)
3.69+1.61
8.86+3.12
3.40
8.46+12.77
Attention
3.60:E1.59
8.77
3.22:11.55
8.36
CloudLSTMM (KC 9)
M AUGMENTING FORECASTING WITH SEASONAL INFORMATION
Finally, notice that seasonal information exists in the mobile traffic series, which can be further exploited to improve the forecasting performance. However, directly feeding the model with data spanning multiple days is infeasible, since, e.g., 7-day window corresponds as input (given that data is sampled every 5 minutes) and it is very difficult for RNN-based models handle such long sequences. In addition. by considering the number of mobile services (38) and antennas (792), the input for 7 days would have 60,673.536 data points. This would make any forecasting model extremely large and therefore impractical for real deployment.
2016-long sequence
To capture seasonal information more efficiently, we concatenate the 30 minute-long sequences (sampled every 5 minutes) with sub-sampled 7-day window (sampled every 2h). This forms an input with length 90 (6 84). We conduct experiments of the mobile traffic dataset (City 1), and show the forecasting performance without and with seasonal information (7-day window) in Table 7 By incorporating the seasonal information, the performance of most forecasting models is boosted. This indicates that the periodic information is learnt by the model, which helps reduce the prediction errors. However, the concatenation increases the length of the input, which also increases the model complexity. Future work will focus on a more efficient way fuse the seasonal information, with marginal increase in complexity.
randomly selected subset (100 antennas)
22


Reviews:
[{"review_id": "rJxT5QF55S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=========== Update after rebuttal\n\nThanks for the clarifications and the update. I recommend acceptance of the paper and updated to 8.\n\nLast comment: please still improve the appearance of Figure 4 by using a more diverse set of marker shapes as well as overlay and offset tricks -- see https://www.cs.ubc.ca/~schmidtm/Software/prettyPlot.html for an example. \n\n============\n\nThis paper introduces a new convolution operator (D-conv) specifically tailored to model point-cloud data evolving over time, i.e. a set of n points with features and localization-coordinates that can evolve over time. The main idea is to use the k-nearest neighbor structure for each point to get a fixed size k window to use in the convolution to determine the new location and feature values of a point (and a permutation-invariant operation). The D-Conv operator is included in a LSTM architecture (CloudLSTM) to enable the spatio-temporal modeling of point-cloud data, and can be combined in standard neural network architectures such as a Seq2Seq with attention. This is in contrast to previous approaches which modeled the data on a grid through preprocessing, or did not include the temporal component for cloud data. Experiments is conducted on 4 benchmark datasets, covering two point-cloud stream forecasting applications, showing how CloudLSTM give lower prediction error than numerous baselines and alternatives.\n\nWhile the D-Conv idea seems fairly simple and natural, it is novel AFAIK and fairly appropriate to model point-cloud data streams. The approach is well situated in the literature, and the experiments are indicative that this method can improve on the current approaches. I am thus leaning towards accept.\n\nThe paper is fairly clear, though the notation is a bit confusing and somewhat sloppy (see detailed comment below).\n\nImportant clarification requested: the current notation suggests that each channel could have a different location for a point p_n, the K nearest points seem to be defined irrespective on the channel. So is the location fixed across channels; or does this paper allow the neighborhood structures to vary across channel?\n\n== Other detailed comments ==\n\n- p.3 Q_n^K -- it seems it would be more appropriate to define it as an ordered list of k points (rather than a set, as this would loose all information about the order); unless you append a new dimension to each point where you put the ordering information there for the purpose of defining the k points in Q_n^K.\n\n- (2) the notation is a bit weird and overloaded for the summation (without being defined). Examples include \"i in U1\" (when U1 is an integer, not a set); \"p_n^k in Q_n^K\" when p_n^k does not appear in the summation (a clearer alternative would be using the notation v(p_n^k)_i^h for the h^th value of channel i of point p_n^k, e.g.; now p_n^k would indeed appear in the expressoin); \"v_n^h in v_n\" -> why not just summing over h as it is really doing? Etc.!\n\n- (2) S_out^j: each p_n^' should be a *tuple* (not a set like currently written).\n\n- Figure 4: the lines are really hard to distinguish just by the similar colors -- please use different markers for the different lines (and offset the marker so that they can be seen)\n\n- Several neighborhood sizes are experimented with. Note though that smaller neighborhood sizes are just *special cases* of bigger neighborhood sizes (by using zero weight on the last few neighbors in the convolution). Wouldn't it make sense to use a big neighborhood size and regularize in some way the weights for the further neighbors?\n\n- Table 2: for SSIM, there are two rows with 0.69 +/- 0.07 (minimal value) -- they could be both bolded.\n\n- Appendix B, they claim that the complexity of finding the K nearest neighbors (in dimension L for n points) is close to O(K L log(n)) if using KD trees. I vaguely recall issues in high dimension though (in particular that the above complexity is only valid for specific distributions of points in low dimension). E.g. see https://en.wikipedia.org/wiki/K-d_tree#High-dimensional_data where it is mentioned that L << log(n) is normally needed to guarantee efficiency. The claim should properly be nuanced.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeeBVfN9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a new convolution operator, named dynamic post-cloud convention over spatiotemporal data, and the convolution operator can be embedded in different neural network architectures, like recurrent neural networks. In order to achieve the convolution over point-clouds by using both value features and the spatial-features, given a data point, the convolution is conducted over its k-nearest neighbors generated by CNN.  They compared the proposed convolution method by embedding it into RNN, GRN, and LSTM against a number of existing methods on two datasets, in terms of MAE, RMSE, PSNR, and SSIM. Overall, this paper is interesting but needs some clarifications on \n\n1. Given that the proposed convolution operator use KNN to choose the nearest neighbors. It would be good to empirically to study how K would affect the performance, does it data-dependent\n2. Is it possible to study the time complexity for various models?\n3. Table 1 and table 2 seem to show that the proposed convolution operator contributes to the performance in terms of the mean of each metric. It might be good to do further oblation test to study which mechanism actually contribute to the performance. The choice of RNN, attention, or the new operator? Furthermore, the std is quite large, which makes one wonder if the improvement is statistically significant.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BkgVBUphtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review Summary\n--------------\nOverall this is almost above the bar for me to accept, but I think there's enough concerns about the method and experiments that I'm hesitant. Strengths include the invariance to point cloud order and the relative simplicity of the architecture (compared to PointCNN). Weaknesses include a vulnerability to outliers, experiments that don't seem to think about practical effects like day-of-week in forecasting, and experiments that leave out baselines to help directly assess the impact of neighbors.\n\n\nPaper Summary\n-------------\nThe paper develops a new neural net architecture for processing data structured as spatial point clouds that vary over time (e.g. hourly traffic at several antennas spread throughout a city).\n\nThe core of the approach is a new neural net unit: the \"D-Conv\" operator (See Eq. 2). The output value at each point is obtained via a weighted combination of nearby coordinates and features, using only the K-nearest neighbors (stored in ranked order) to maintain invariance to the original order of points. This layer can be included in modern convolutional (CloudCNN) or recurrent (CloudLSTM) or attention-based architectures in a straightforward way.\n\nUnlike many previous methods that require converting point clouds to quantized regular grids, the present approach directly consumes point cloud data. Unlike some existing methods like PointCNN, it avoids information loss (does not reduce dimension from input to output layer).\n\nTwo experimental evaluations are conducted: forecasting mobile app traffic across 2 European cities (given past 30 min, predict next 30 min), and air quality across several regions in China (given last 12 hrs, predict next 12 hrs). In both experiments, the locations of the sensors are fixed across time. Fig 4 further looks at traffic forecasting as a function of the lookahead time, from 0-3 hours ahead.\n\n\nNovelty & Significance\n-----------------------\n\nThe paper definitely tackles an important problem (point cloud forecasting). \n\nThe present paper's new \"D-Conv\" operator appears new, though it looks really like a simplification of the PointCNN's \"X-Conv\" operator rather than a brand new operator. \n\nThe most similar work seems to be the PointCNN (Li et al NeurIPS 2018). This work's contribution was a new \"X-Conv\" operator, which also consumes point clouds and produces learned representations. X-Conv, like the present paper's D-Conv, computes K-nearest neighbors of each point p, but performs first an embedding of each neighbor to a learned \"local\" feature space and then performs convolution on this embedding. Perhaps the biggest practical difference is that D-Conv has fewer parameters (does not perform the embedding) and does not reduce dimensionality from input to output. \n\nTechnical Concerns\n------------------\n\nMy biggest concerns are that the D-Conv has a strong reliance on nearest neighbors. This means the D-Conv has not much accomodation for \"outlier\" points that are far from others. The X-Conv operator has some nice properties in this regard (it changes coordinate systems so neighbor locations are centered around the current point), but I don't see this in the D-Conv operator, as in Eq. 2, where the coordinate locations are fed directly into the weighted sum after global rescaling to (0,1). I would imagine that data with outliers (whose values are unlike most others) would dramatically hurt performance, as the weights of D-Conv would need to be shared equally by outliers and inliers.\n\n\nExperimental Concerns\n---------------------\n\nIs there a good reason to not try to compare on publicly available datasets like those used in the PointCNN paper (focusing only on the non-temporal versions of the model)? Using proprietary datasets makes following up on this work a bit hard, would be nice to have some reproducible experiment.\n\nIt's not clear to me that the experiments here consider realistic scenarios. Why would I predict mobile app traffic using only the past 30 minutes of data? Why predict air quality using only the last 12 hours? Certainly there are time-of-day, day-of-week, and seasonal effects that are all important. At a minimum, I'd think that for the mobile traffic case you could at least look at consuming the last 48 hr of data and predicting the next 30-90 minutes. I suspect that would make even simpler models do much better.  \n\nFurther, I think the experiments are missing some key simple baselines (or I misunderstand something). For example, rather than the complicated CNN/LSTM architectures, why not try to directly see how much value there is in \"neighbors\" in this 2d space? At each point, you can make predictions using only the K nearest neighbors' data, with K swept from 1 to 100 or something. I would expect with these features, using just a simple MLP or RNN would do quite well. I'd like to see a stronger qualitative case made for why we expect the complicated DConv weighting operator here to do better than this baselines.\n\nOverall, the results tables appear promising (for app traffic forecasting in Table 1, the proposed CloudLSTM achieves 3.66 MAE compared to 4.95 for PointCNN and 4.8 for an MLP). However, it's not clear why and I'd like to understand why. Is it that the other approaches are overfitting? \n\n\nMinor Concerns\n--------------\nI would suggest avoiding calling the method \"\\mathcal{D}-Conv\", and instead use just \"DConv\", since this is easier to type into search engines and easier to search for in a PDF document\n\nRelated: Point clouds could be represented as graphs, and then use graph embeddings as feature representations", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an approach to forecasting over temporal streams of permutation-invariant data such as point clouds. The approach is based on an operator (DConv) that is related to continuous convolution operators such as X-Conv and others. The reviews are split. After the authors' responses, concerns remain and two ratings remain "3". The AC agrees with the concerns and recommends against accepting the paper.
Decision: Reject