Spatio-temporal Disentangled representation learning for mobility prediction
Authors: Sichen Zhao
Wei Shao
Jeffrey Chan
Flora D. Salim
Venue: ICLR 2022 Submitted (2022)
Keywords: Disentangled representation learning
Spatio-temporal data
principle of relevant information

Abstract: Spatio-temporal (ST) prediction task like mobility forecasting is of great significance to traffic management and public safety. 
There is an increasing number of works proposed for mobility forecasting problems recently, and they typically focus on better extraction of the features from the spatial and temporal domains. Although prior works show promising results on more accurate predictions, they still suffer in characterising and separating the dynamic and static components, making it difficult to make further improvements. Disentangled representation learning separates the learnt latent representation into independent variables associated with semantic factors. It offers a better separation of the spatial and temporal features, which could improve the performance of mobility forecasting models. In this work, we propose a VAE-based architecture for learning the disentangled representation from real spatio-temporal data for mobility forecasting. Our deep generative model learns a latent representation that (i) separates the temporal dynamics of the data from the spatially varying component and generates effective reconstructions; (ii) is able to achieve state-of-the-art performance across multiple spatio-temporal datasets. Moreover, we investigate the effectiveness of our method by eliminating the non-informative features from the learnt representations, and the results show that models can benefit from this operation.

Under review as a conference paper at ICLR 2022
SPATIO-TEMPORAL DISENTANGLED REPRESENTATION LEARNING FOR MOBILITY FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
Spatio-temporal (ST) prediction task like mobility forecasting is of great signif- icance to traffic management and public safety. There is an increasing number of works proposed for mobility forecasting problems recently, and they typically focus on better extraction of the features from the spatial and temporal domains. Al- though prior works show promising results on more accurate predictions, they still suffer in characterising and separating the dynamic and static components, making it difficult to make further improvements. Disentangled representation learning separates the learnt latent representation into independent variables associated with semantic factors (Duan et al. 2019). It offers a better separation of the spatial and temporal features, which could improve the performance of mobility forecasting models. In this work, we propose a VAE-based architecture for learning the disen- tangled representation from real spatio-temporal data for mobility forecasting. Our deep generative model learns a latent representation that (i) separates the temporal dynamics of the data from the spatially varying component and generates effective reconstructions; (ii) is able to achieve state-of-the-an performance across multiple spatio-temporal datasets. Moreover, we investigate the effectiveness of our method by eliminating the non-informative features from the learnt representations, and the results show that models can benefit from this operation.
INTRODUCTION
-patio-temporal prediction tasks like mobility forecasting are critically important for smart city applications (Zheng et al. 2014). With the help of the rapid deployment of 1oT (Internet of Things) devices and sensors, massive amounts of saturated ST datasets are available, and many researchers have put their efforts to improve the performance of ST prediction (Zhang et al., 2016 Wang et al.) 2018, Zonoozi al., 2018 Jin al.) [0188] Yao et al., (0196). A central problem deep learning for crowd flow prediction tasks is the extraction of features from both spatial and temporal domains. Researchers first tried to extract the entangled spatio-temporal features directly from the data (Zhang et al. 2016 Then, they tried extract features from spatial and temporal domains separately. [Wang ct al. (2018) and Zonoozi et al. (2018) proposed models that extract periodic representations or short-term temporal features directly from the ST data using recurrent-based convolution operations and find their effectiveness in producing more accurate predictions. Based on that, in STDN (Yao) cet al. (011b)). the explicit extraction of the long-term periodic information was shown to also improve the prediction.
Although the existing approaches appear be powerful in terms of results and predictions. two major challenges hinder these models: 1) Difficulty in characterising dynamic and spatial components. Spatio-temporal correlation is more complex since it comprises dependencies from both the spatial and temporal regions. Models like LDRSN (Tian et al., (2020) and Region'rans (Wang et al.) 2018)) are capturing temporal dependencies explicitly. However, their complicated structure makes it hard to characterise and validate the effectiveness of their dynamic and spatial components. 2) Difficulty in separating the extraction of spatial and temporal features. Except for the spatial and temporal features from the most recent data, many approaches try to extract long-term/periodical features to improve the performance. However, without an explicit separation mechanism, the extracted long-term temporal dependencies introduce irrelevant noises to the predictor (factors do not vary with time and are only relevant to the long-term sequence). The extent to how generative models can model, extract and disentangle the spatial and temporal features in ST-raster data is open problem.

---

Under review as a conference paper at ICLR 2022
To address the above challenges, we introduce the disentangled representation learning to the mobility forecasting task. Disentangled representation learning, which separates the learnt representation into independent variables such that each variable relates to one semantic factor of sensory data (Bengio et al., 2013), offers a solution for the problem mentioned above. For a spatio-temporal task like mobility forecasting, an ideal disentangled representation should have the ability separate time-relevant components from the factors that don't vary with time, which can help improve the predictor's performance.
Many prior works have explored disentangled representation learning for spatio-temporal data (Hsu al. 2017: Li & Mandi 2018 Denton & Birodkar 2017: Zhu al., 2020). Most of them assume that the features can be disentangled into two sets: a set of dynamic features that extract the temporal correlations and a set of static features that describe factors that are constant through the input sequence. However, since most of them focus on the movement of some predefined objects, their assumption is inaccurate when applying to datasets for mobility forecasting. For example, a sudden accident might show no relationship to the temporal influence from the previous timestep and the static traffic network structure. Therefore, there are still gaps in how to model the spatio-temporal data using the disentangled representation learning method. To address this problem, each timestep has its own temporal features and features that do not vary with time. By doing so, our model can better formulate the spatio-temporal data, and we can achieve controlled data generation frame by frame.
assume that
In this work, we proposed VAE-based model learn disentangled spatio-temporal representation. Compared to the conventional methods for the ST prediction problem, our approach extracts the entangled features first and then explicitly separates them into temporal variables and spatial variables using disentangled representation learning method. This will force the model to keep the learnt spatial/temporal features as mutually exclusive as possible. For applying disentangled representation learning method to real ST data, we assume that each frame has its own spatial and temporal variables and separate these two groups with auxiliary regularisation. It helps the model to formulate the complicated spatio-temporal sequence. Our experimental results (see Section + show that the learnt representations have a similar level of performance with the current state-of oftt--ofthhearr methods. Our key contributions can be summarised as follows:
1. We propose novel approach to learn disentangled spatio- representations for mobility forecasting tasks. The learnt representation is separated into two independent groups: spatial and temporal factors.
2. We conducted several experiments multiple spatio-temporis datasets and used the learnt representation for mobility forecasting. Results show that our methods achieve state-of-the- art performance compared to other baseline mobility forecasting methods.
3. We investigate the effectiveness of our methods under the "Closeness, Period, Trend" scheme and how to further improves the model's 's performance by selecting the informative features from the learnt representations.
RELATED WORK
Deep spatial-temporal networks for mobility forecasting: In order to make accurate traffic pre- diction, many researchers have paid attention to capturing the spatio-temporal dependencies hidden behind the traffic data. Besides the conventional methods like Seasonal ARIMA (Moreira-Matias et al., 2013). deep learning methods are increasingly used in more and more works for mobility forecasting.
ST-ResNet (Zhang et al. 2016) was proposed to capture the spatial dependencies through a stack of residual convolution layers. also stacks the frame from a near and a distant time period to capture temporal dynamics. The goal of using the residual units is to overcome the gradient vanishing problem, and the results show that capturing distant spatial features can improve the performance of mobility forecasting, DeepSTN+ (Lin et al. 2019) proposed a ResPlus unit that is capable of capturing long-range spatial correlations. Although they succeed in extracting distant spatial features, they lack attention capturing the temporal dependencies. To further explore the effectiveness of temporal features, recurrent- seuureent-basdd approaches were introduced to capture the temporal correlations. PCRN (Zonoozi al. 2018) was proposed, which first extract entangled spatio-temporal representation
2

---

Under review as a conference paper at ICLR 2022
using convolutional recurrent network (CRN) and then updating the periodic representations by CRN's hidden state. Attention-base LSTM methods are adopted by STDN (Yao et al. (019b) and ST-DCCNAL (Li et al. 2019), which try to capture long long-term temporal dependencies. In summary, our proposed method differs from other methods in the explicit separation of spatial and temporal features using a disentangled representation mechanism.
Most prior works on the dis- entangled representation learning problem are developed based on Variational Autoencoders (VAE) (Kingma & Welling 2013), which is an unsupervised generative learning method. 3-VAE, proposed by Higgins et al. (2016), forces the inference model to disentangle the latent representation by adding a new hyperparameter 3 to create an information bottleneck on the prior. FactorVAE (Kim & Mnih 2018) further breaking down the objective function and try to enhance disentanglement by penalising the total correlation of the learnt representation. As for sequence modelling, number of prior publications have extended VAE to video and speech data (Fabius & Van Amersfoort [014] CChung 2015 Bayer & Osendorfer] 2014). These models, although being able to generate realistic et al. sequences, do not explicitly disentangle the representation of time-invariant and time-dependent information. Thus it is inconvenient for these models to perform tasks such as controlled generation. S3VAE (Zhu et al.] (220) is proposed to separate static and dynamic factors of sequential data. Another approach proposed by (Li & Mandt (018) is also focusing separating the dynamic factors from static factors. Although they share a similar idea which uses an RNN-based architecture to extract dynamic factors for each timestep. they use different prior setups. Each frame in (Li & Mandt 2018) has its own content features while the time-invariant variables are shared by the whole sequence in S3VAE. As for the spatial- temporll type of data like video, SV2P (Babaeizadeh et al. 2017) uses the variational model to extract the time-nnvriin latent and make predictions for multple frames. Models like (Denton& & Birodkar 2017 Hsieh et al., 2018) try to factorise each frame into a stationary part and temporally dynamic component.
Disentangled Representation Learning for Spatio-temporal data:
3 SPATIO-TEMPORAL VAE MODEL
3.1
DATA
In this work, we will use D (X denote spatio-temporal raster dataset that comprises N i..d. sequences. Each X = x1.7 = { x2,.... zT in that dataset denotes a sequence of raster data with T frames. Since we focus on mobility forecasting tasks using grid-based spatial representations, which all have the similar dimensions H x W. Hence each frame 2t E IR HWW represents the mobility flow of a certain area at a given time interval
3.2 SEPARATING SPATIAL FEATURES FROM TEMPORAL FEATURES
Many prior works have explored disentangled representation learning for spatio-temporal data. Models like FHVAE (Hsu et al. 2017) tried to separate global variables from segment (dynamic) variables for speech data. DSVAE (Li & Mandi 2018), DRNET (Denton & Birodkar] (017) and S3VAE (huu et al. 2020 aim at the disentangled representation for video, which also factorised latent variables into static and dynamic parts. However, current approaches for learning the disentangled representation on sequence or spatio-temporal data like video assume that there is a fixed content or object shared by all frames in the sequence since they are using video datasets like the Stochastic Moving MNIST (Denton & Fergus] 2018) and Sprite (Li & Mandt, 2018). Sequences in these datasets often comprise images describing the movement of set of the same numbers or virtual avatars. Under this context, static features describing this object can be extracted, and its performance can be evaluated by the accuracy of a classification task. However, for most of the real apatio-tempont datasets, the static features are not enough. For example, in a traffic flow dataset, in addition to the fixed structure of the traffic network and the temporal influences, there might be some hotspot suddenly emerges. At that specific time, the rise of those events shows no correlation to the temporal features and the fixed content. Therefore, we assume that for each frame, data is generated based on two sets of features: set of temporal features which comprise the influence from the sequence before it and a set of spatial features that describe the structure of the network and local events. In this work, propose novel architecture that extracts mixed (entangled) feature maps for each timestep in the input sequence and then separates them into temporal and spatial variables. allows us analyse and controlled generate each frame separately.
3

---

Under review as a conference paper at ICLR 2022
X
Tomparal Karving Compunin
Carns Meck
Gote po4
Sure Citwer
Care deck
Imagebnooder
Are
Secosee
Carw Deck
Sone Soce
X
Feature Modile
Feature
Spatial Verying Camponent
(b) A detailed structure of the Fea- ture Separation module
(a) The framework of our proposed model
Figure I:
The framework of our proposed model. Each frame of a fed into the Image Encoder first extract entangled high-level feature maps, which are then passed through a temporal gated convolution layer to capture the time-varying variables (temporal features) _T'e
sequence 21:T is
21:7" Then the time-irrelevant variables (spatial features) 25" for each frame Ie is captured through several convolutional blocks. In addition the objective of the VAE, Total Correlation regularizer used to encourage the disentanglement of the learnt representation. Reconstruction input is generated for each frame based on the sampled latent from both domains.
The architecture of our method for spatio-temporal disentangled representation learning is illustrated in Fig Tal It consists of three major modules: (i) Image Encoder to yield initial feature maps from the input spatio-temporal sequence. (ii) A Feature Separation module to separate the spatial and temporal variables from the initially mixed feature maps. (iii) A Decoder to reconstruct the sequence frame by frame based on their corresponding spatial and temporal variables.
The Image Encoder comprises several layers of CNN, and the feature maps extracted from each frame separately. The goal here is to yield some higher-level features for the next step. An Instance Normalisation layer is used after each convolutional layer to improve the reconstruction results since using batch normalisation might remove the eettance-specffe contrast information from the data, which is useful in the later step (Ulyanov al., 2016).
Intuitively, without any disentanglement constraints, the feature maps that come out of the Image Encoder should be the entanglement of features from both spatial and temporal domain. Therefore, the goal for the Feature Separation module is to separate them explicitly. For temporal features, inspired by Yu et al., 2017; Gehring et al.) 2017), we use a Gated Convolutional structure to capture temporal dynamics from the mixed feature maps. This temporal gated convolution layer contains two 1-D causal convolution layers. As Fig Tb shows, those 1-D convolution operations will be applied on the time axis for each pixel of the feature maps, which means the capture of the temporal dynamics from each mixed high-level feature. is much easier to understand the usage of the Image Encoder here since the 1-D convolutional operation on the high-level features map is definitely better for separating the temporal dynamics than applying them on the individual pixel of the original input.
The Decoder module is used to reconstruct each frame 2, by using their corresponding spatial variables 2 and temporal variables 2Te input.
3.3 SPATIO-TEMPORAL VAE MODEL.
Priors: In this work, we proposed a spatio-temporal variational autoencoder architecture to learn disentangled representations from ST raster datasets. Our assumption for variational autoencoder is that each input is generated from a corresponding latent representation which can be separated into two disentangled subgroup: variables z S8 which contains the spatial (time-ireleevant) features and the temporal (limevvarying) features =f.
On the one hand, since the spatial variables z SSp ~ No0, 1) and the spatial variables for the whole On the other hand, the prior of the temporal dynamic
_S is considered time-irrelevant therefore its prior is
defined as standard Gaussian distribution sequence can be formed = -Ith =t
4

---

Under review as a conference paper at ICLR 2022
variables follow sequential prior zf. 2. The distribution of this sequential prior is defined as =/" N (Pe, diag (oi)) where [u,, oi de (zãt). The mean b4 and variance of are parameters that conditioned on all of its previous temporal variables 2 and can be parameterised by the gated-convolution layer. Thelatent prior that combines both the spatial and temporal variables is formed as:
p(z,) p(z
((=;"))((1)) (=(11))))))))))))))))))))))))) p(=
P =1))
Generative model: For the generative model, we assume that the generation of each frame 2, at given time depends on the combination of its corresponding spatial variables 2,50 and temporal variables z Therefore, the generation process for the single frame in the whole sequence I:T can be formed as:
T
Po( 21) (0 (2(el22))((2)) po(2))2))p
Sp
II
where are the parameters for decoder.
Inference models: We use deep structured model encoder to approximate the posterior distribution, which can factorise the latent into disentangled spatial and temporal components. The amortised variational distribution is formed as:
T
Te Sp 21:T 2T ((1IT)
= II do t=l
2t
Sp 14
II do (2/ (x())))))) 1=1
(3)
3.4 LOSS
In this work, the objective of our proposed method is defined as the combination of the VAE loss and total correlation regularisation. It can be formulated as: L Lv + 3LTc where 3 is the hyperparameter for the TC regularisation. Theoretically, applying higher value on 3 will emphasise disentanglement of the learnt representation and lead to better separation the spatial and temporal variables. We estimate the objective based on Factor approach (Kim & Mnih 2018).
VAE Objective Function: The objective function of the our method derived from the variational lower bound (Evidence Lower Bound, ELBO) of the vanilla VAE (Kingma & Welling 2013) and is formed as follow:
T E D<rl(a(?""<)))))(<((
(0A5(0,0501.1) = E
log pa (2 1 x[])
90 (x)r))
T
Σ
Dkilgs(z ( pp Te (2)
(4)
t=1
Note that S3VAE (hhu et al. 2020) already propose a sequential VAE that consider the continuity of dynamic variables, but only assumes that there are whole set of statice features share by the whole sequence. In contrast, model the temporal and spatial features for each time step independently, resulting in detailed information for emergencies mobility forecasting data.
Total Correlation Regularization: To encourage the overall disentanglement of the learnt represen- tation, we introduced the Total Correlation (TC) among variables as regularization term. quantifies the dependency among variables (Alfonso et al. 2010). Experimental results from 3-TCVAE (Chen et al.) 2018) and Factor VAE (Kim & Mnih] 2018) show that, by amplifying the penalty on this term, the dependence between the variables is reduced hence emphasising the disentanglement.

---

Under review as a conference paper at ICLR 2022
In this work, estimate the total correlation using the same approach like FactorVAE (Kim & Mnih 2018), i.c. by introducing a discriminator and using the independence testing trick and the density-ratio trick approximate the KL term in the above equation.
4 EXPERIMENTS
4.1 DATASET AND METRICS
In this work, focus learning disentangled representation of spatio-temporal raster data. There- fore, we choose to conduct the experiments on the following three real-world urban flow datasets:
BikeNYC (Lin et al. 2019) a bike usage data collected from New York City's Citi Bike bicycle sharing service, which records the trajectory of all shared bikes in the system. This work covers the time period from 201 4-0-001 to 2014-09-30.
TaxiNYC Yao et al.
2019a) is a dataset that contains taxi in-out flow data taxi New York City, Taxi GPS data, which covers the period from 2015-01-01 to 2015-03-01.
created from the NYC-
TaxiBJ (Zhang et al. 2016) comprises the taxi in-out flow data that aggregate the taxi GPS position Beijing from 2013 to the year 2016. Although span across 4 consecutive years, the data not continuous (covers the period: 2013-07-01 to 2013-10-30, 2014-05-01 to 2014-06-30, 2015-03-01 to 2015-06-30, and 2015-11-01 to 2016-04-10).
For disentangled representation learning, use all of the data to train the feature separation module, and for the mobility forecasting tasks, we use the same setup described (Xue & Salim, 2021). i.c., the first 80% data is used for training the prediction model, and the rest 20% data is used for testing. Besides, Min-Max normalization is adopted to transform the urban flow values into the range [0, 1] for better training purpose. We evaluate the effectiveness of our models with two commonly used metrics: the Mean Absolute Error (MAE) and the Root Mean Square Error (RMSE).
4.2 IMPLEMENTATION DETAILS
To compare the performance our model in mobility forecasting with the current state-of- the-art methods, we use the "Closeness, Period, Trend" scheme to form the input of our predic- tion model. widely used in the urban flow prediction area, where all three of them com- prise sequence of raster data. It designs a set of unique input sequences, namely Closeness, Period, and Trend, which correspond to the re- cent time intervals, daily periodicity. and weekly trend, respectively (Zhang et al., (2066). Those three sequences are then fed as the input their models.
Peaiicion
Last Friday (1608)
Transde (PN00)
Fielry powne
Tiowntergge modkee
Daretingg modiue
MELP
As shown in Figure 2 in this work, same setup used by VLUC-Net (Jiang et al.) 2019). where the Closeness sequence contains the previous six steps before the prediction tar- get; the Period the sequence of the previous day, and the Trend comprises data from the pre-
the
Figure 2: Illustration of the "Closeness, Period, Trend" components and the actual architecture in the mobility forecasting experiment.
vious week. We first train our model to learn disentangled representation on the closeness, period and trend sequences separately. Then the learnt representations are fused train Multi-Layer perceptron (MLP) regressor to predict the next frame of the sequence. It should be noted that the input of the MLP is not the actual features 2, but the learnt μ and a of the distribution. The logic of doing this is because the sampling part will introduce uncertainty which has a huge impact on the training.

---

Under review as a conference paper at ICLR 2022
Table 1: Effectiveness Evaluation of Traffic In-Out Flow Prediction. Lower is better.
BikeNYC
TaxiNYC
TaxiBJ
RMSE
MAE
RMSE
MAE
RMSE
MAE
HA
4.874
1.500
21.535
7.121
45.004
24.475
CNN
4.511
1.574
16.741
6.884
23.550
13.797
ConVLSTM
3.174
1.133
12.143
4.811
19.247
10.816
Zhang
2016
3.191
1.169
11.553
4.535
18.702
10.493
DMVST-Net Yao et al.
2018
3.521
1.287
13.605
4.928
20.389
11.832
D Lin et al. 2019)
3.205
1.245
11.420
4.441
18.141
10.126
STDN Yao et al.]
9a)
3.004
1.167
11.252
4.474
17.826
9.901
ILUC-Net(Jaag et al. 2019
3.119
1.124
10.654
4.157
18.378
10.325
Without explicit
5.412
1.537
19.985
6.281
21.631
12.202
disentanglement
Spatial Only
3,584
1.285
14.640
4.812
21.111
12.435
Temporal Only
3.107
1.167
14.175
4.669
19.825
11.638
Ours
2.903
1.119
12.022
4.055
19.185
10.741
4.3 MOBILITY FORECASTING: COMPARISON AGAINST OTHER METHODS
In our experiments, we compare our method against the following mobility flow prediction methods: Historical Average (HA); Convolutional Neural Network(CNN); Convolutional LSTM (ConvLSTM);
ST-ResNet(Zhang al. 2016); DMVST Yao al. 2018;; DeepSTN++Lin al. 2019); STDN Yao et al. 2019a); VeUn-Net(Jangg et al.) 2019).
The overall evaluation results on effectiveness are summarised in Table for TaxiBJ, TaxiNYC and BikeNYC. The upper half of Table shows the results of baseline methods on those datasets, and the lower part shows the results of our proposed approach. The best result for each column given in bold, and methods except HA, CNN, and ConvLSTM are all the current state-of-the-ar methods in crowd flow prediction. In general, we can find that our method shows the best results on the BikeNYC and TaxiNYC datasets and compatible performance on the TaxiBJ dataset.
Besides the crowd flow prediction with a full feature set, we also trained models with only spa- tial/temporal features to see which set contributes more to the mobility forecasting. We find that the prediction performance of the models trained on the temporal feature set leads those trained on the spatial features by relatively large margin. This shows that for the crowd flow prediction task, the extraction the temporal dependencies is important than the spatial dependencies, which also coincides with the research direction in this area. It should also be noted that the models trained on the temporal feature already show compatible results with the current state-of-the-as methods. The still room for improvement since the input of our method is the raw data alone with no context information.
4.4 ABLATION STUDY
To explore the effectiveness of each module in the "Closeness, Period, Trend" scheme, we perform an ablation study consider different configurations of the input sequences. The detailed configuration and their corresponding results are summarised in Table The left half of the table shows the sequences used for each setup. By discarding extracted features from some sequences in the scheme, we are able to evaluate their effectiveness in regrading the mobility forecasting task. Since the quality of the learnt disentangled representation is sensitive to the choice of hyperparameters (Duan et al.) 2019), trained multiple feature separation modules with different hyperparameters. The mean and standard deviation of each configuration on all three datasets are presented in the right half of Table 2
We can first find that the performance of models using period or trend sequence alone is poor, and sometimes adding those extra information does not help the mobility forecasting task. On average, the configuration that achieves the best results for all three datasets is not configuration C6. Although the features from the trend sequence improve the results for the BikeNYC dataset by a tiny margin, introducing the middle/long-temm data worsens the performance for both the TY and TaxiBJ
7

---

Under review as a conference paper at ICLR 2022
Table Seven different configurations of the input sequences and their corresponding RMSE and MAE results. It should be noted that C6 is the widely-used setup of the current state-of-the-an mobility forecasting models
Configuration
BikeNYC
TaxiNYC
TaxiBJ
Closeness
Period
Trend
RMSE
MAE
RMSE
MAE
RMSE
MAE
3.660
1.209
14.323
4.670
22.035
14.078
CO
X
x
11.88
10.45
+1.31
10.35
16.87
+3.57
5.924
1.658
25.364
7.268
35.222
21.785
C1
/
X
+2.12
+0.61
+2.06
400.5
+.3.22
+2.24
4.217
1,361
22.387
6.841
42.361
25.967
C2
x
/
+2.17
+0.53
+2.58
+0.70
+2.93
+185
3.707
1.222
14.431
4.704
22.140
14.182
C3
V
X
+2.43
+0.54
+1.6
10.44
+..6
+3.52
3.558
1.198
17.020
5,281
22.585
14.434
C4
X
/
=1.77
+0.44
+2.38
10.56
#7.21
#3.66
4.406
1.388
22.634
6.692
36,454
22.492
C5
/
1.92
10.49
+2.34
10.54
33.24
1192
3.601
1,210
16.677
5.173
22.504
14.388
C6
12.10
+0.50
+2.29
+0.52
16.97
+3.52
datasets. One of the reasons might relate the size the representation. Although there might be some information that can contribute to the prediction, using all three sequences tripling the dimension of the input and the noise offset the potential benefit. Besides, we can find out that the variance results are high for all three datasets, which agree with the assumption that the quality of the learnt representation regarding the mobility forecasting task is sensitive to the choice of hyperparameters.
4.5 INFORMATIVE FEATURES
Given the thought that the dimension of MLP's input might be too large and the learnt representations from the previous section, we want investigate whether can find the "informative" features from the learnt representations. And for those "normal" representations (contains valuable information but does not achieve the best results), will the models benefit when we exclude those "non- -nfoomnrrrmatve"" features from their input. In our work, we define the "informative" features using the definition from Unsupervised Disentanglement Ranking (UDR)(Duan et al., (019);: A latent dimension is treated as an "informative" feature if it learns a latent posterior that diverges from the prior.
{
1
K (=(gg(tal=)) > 0.001 otherwise
(a)
(5)
where a is the index of the variable. We is larger than the UDR (Duan et al. 2019). To avoid the influence from random noise, we test the performance of mobility forecasting on all representations that we get from the previous section, and the results are summarised in Figure 3a
smaller threshold here since the number of features
First, it should be noted that there exist representations that do not contain any informative features. As shown in the figure, those "poorly-leamt" representations perform a lot worse compared to the ones with informative features. That proves that ""well diell-dsentangled, well-learnt" representation can contribute to the downstream task, at least for the mobility forecasting task. To prove the necessity of the filtering operation (remove the nonniinformative features), we first decrease the representation size and train the feature separation module to extract representations. The results show that when the size is below a certain threshold, all representations with smaller sizes will not contain any informative features. And the ones containing informative features will always have some space. That indicates the necessity of the filtering operation since we still haven't 't found a solution to learn disentangled representation that only contains informative features.
Another thing that we want to investigate is whether the models can benefit when removing those non-informative features from their input. Therefore, train those MLPs with only the informative
8

---

Under review as a conference paper at ICLR 2022
RIkNYC
TaxiM/ Informative
TaxiFul
22
Informative
Informaivv
Todon Irue
true
1.8
25
NAL
1.4
12
Configuratin
Configuaaiion
Configurstin
(a) Results of different configurations for representation contain/not contain informative features.
0.5
Metro
MAE
Metro MIE
Metic
MME
RMGE
NMGE
rusc
02
Improvemont after
-2.1
Sel Contiguration
Configuratinn
Set Configuration
(b) Improvements of different configurations after only using informative features input.
Figure 3: The boxplot of RMSE and MAE results for different representations (learnt with different hyperparameters).
features and calculate the difference in performance and summarised the results in Figure 3b] It is worth noted that the MLPs have the same structure except the first layer due to the reducing amount of input features.
In summary, for the BikeNYC and TaxiNYC datasets, we can find out that, although the MLP that trained with the best representation does not get performance improvement, the majority of the MLPs that trained with "normal" representations show better results after removing the "non-informative" features. As for the TaxiBJ dataset, there a slight drop performance but only with a tiny margin. The reason for that might be due to the different sizes and data distribution of TaxiBJ. The overall grid size for TaxiBJ is larger than the other two datasets, and the Cumulative Distribution Function (CDF) shows that it contains more large values in the dataset, which might lead to different behaviour when we are removing the hon-inforrmative features. More details can found in the Appendix.
5 CONCLUSION
Spatio-temporal (ST) prediction tasks like mobility forecasting have attracted significant attention since they greatly influence traffic management and public safety. We introduce the disentangled representation learning method and modify it to fit spatio-temporal data. The experimental evaluation results show that our method can achieve state-of-the- oftthe-rtttttttt performance and is able extract desirable spatial/temporal features. Moreover, we investigate the effectiveness of pecen//middle//on-ttoe term temporal features and find that sometimes our method can achieve state-of-thed results without long-term temporal features. Finally, we also demonstrated that well-learnt representation shows better results in mobility forecasting tasks and removing the non-informative features from the input of downstream models sometimes can also boost performance. Hence, we hope that our method can contribute to the mobility forecasting task by introducing the disentangled representation learning mechanism. One future direction of this work is forcing the model to learn tation that only contains "informative" features. A better mechanism to separate the spatial/temporal features and link them to real semantics also needed be investigated.
compact represen-

---

Under review as a conference paper at ICLR 2022
REFERENCES
Leonardo Alfonso, Amold Lobbrecht, and Roland Price. Optimization of water level monitoring network in polder systems using information theory. Water Resources Research, 46(12), 2010.
Mohammad Babacizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint 17X.v.1710.11252, 2017.
Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks. arXiv preprint arXiv: 1.7660, 2014.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence. 35(8):1798-1828. 2013.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K. Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances Neural Information Processing Systems, pp- 2610-2620, 2018.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data, arXiv preprint
2015.
Emily Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. arXiv preprint 1705
2017.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In International Conference on Machine Learming, 1174-1183. PMLR, 2018.
Sunny Duan, Loic Matthey, Andre Saraiva, Nicholas Watters, Christopher P Burgess, Alexander Ler- chner, and Irina Higgins. Unsupervised model selection for variational disentangled representation learning. arXiv preprint arXiv:1905.12611 2019.
Otto Fabius and Joost R Van Amersfoort. Variational recurrent auto-encoders. arXiv preprint 1rXiv::412.665 2014.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence sequence learning. In International Conference Machine Learning, pp. 1243-1252. PMLR, 2017.
Irina Higgins, Loie Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
Jun-Ting Hsieh Binghin Liu, De-An Huang, Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. arXiv preprint 046.06660 2018.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data, arXiv preprint arXiv: (70.0070700 2017.
Renhe Jiang, Zekun Cai, Zhaonan Wang, Chuang Yang, Zipei Fan, Xuan Song, Kota Tsubouchi, and Ryosuke Shibasaki. Vlue: An empirical benchmark for video-like urban computing on citywide crowd and traffic prediction. arXiv preprint arXiv: /9/ 1.06982, 2019.
Wenwei Jin, Youfang Lin, Zhihao Wu. and Huaiyu Wan. Spatio-temporal recurrent convolutional networks for citywide short-term crowd flows prediction. In Proceedings of the 2nd International Conference on Compute and Data Analysis, pp. 28-35, 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv: 100.:v8802.0593 2018.
Diederik Kingma and Max Welling. arXiv: 2013.
Auto-encoding variational bayes.
arXiv preprint
10

---

Under review as a conference paper at ICLR 2022
Wei Li, Wei Tao, Junyang Qiu, Xin Liu, Xingyu Zhou, and Zhisong Pan. Densely connected convolutional networks with attention Istm for crowd flows prediction. IEEE Access, 7:140488- 140498, 2019.
Yanjun Li, Shujian Yu, Jose C Principe, Xiaolin Li, and Dapeng Wu. Pri-vae: Principle-of-ellevnn- information variational autoencoders. arXiv preprini arXiv:2007.0663 2020.
Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder. arXiv preprint
arXiv: 02991, 2018.
Ziqian Lin, Jie Feng, Ziyang Lu, Yong Li, and Depeng Jin. Deepstn+:
Context-aware spatial-temporal neural network for crowd flow prediction in metropolis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1020-1027, 2019.
Luis Morcira-Matinn Joao Gama, Michel Ferreira, Joao Mendes-Morudd and Luis Damas, Predicting laxi-passenger demand using streaming data. IEEE Transactions on Intelligent Transportation Systems, 14(3):1933-1422, 2013.
Chujie Tian, Xinning Zhu, Zheng Hu, and Jian Ma. Deep spatial-temporal networks for crowd flows prediction by dilated convolutions and region-shifting attention mechanism. Applied Intelligence, 50(10):3057-3070, 2020.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprini arXiv: 1607.08022 2016.
Leye Wang, Xu Geng, Xiaojuan Ma, Feng Liu, and Qiang Yang. Cross-city transfer learning for deep 2018.
spatio-temporal prediction. arXiv preprint
Hao Xue and Flora D Salim. Termcast: Temporal relation modeling for effective urban flow forecasting. In PAKDD (1), pp. 741-753, 2021.
Huaxiu Yao, Fei Wu, Jintao Ke, Xianfeng Tang, Yitian Jia, Siyu Lu, Pinghua Gong, Jieping Ye, and Zhenhui Li. Deep multi-view spatial-temporal network for taxi demand prediction. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Huaxiu Yao, Xianfeng Tang. Hua Wei, Guanjic Zheng, and Zhenhui Li. Revisiting ppatial-temporal similarity: A deep learning framework for traffic prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 5668-5675, 2019a.
Huaxiu Yao, Xianfeng Tang, Hua Wei, Guanjie Zheng, and Zhenhui Li. Revisiting spatial-temporal similarity: A deep learning framework for traffic prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 5668-5675, 2019b.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. arXiv preprint 170.v:1709.04875 2017.
Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. arXiv preprint 1rXiv:1610.0000 00000,0011,,,, 2016.
Yu Zheng, Licia Capra, Ouri Wolfson, and Hai Yang. Urban computing: concepts, methodologies, and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 5(3):1555, 2014.
Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. S3vae: Self-supervised sequential vae for representation disentanglement and data generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6538-6547, 2020.
Ali Zonoozi, Jung-jae Kim, Xiao-Li Li, and Gao Cong. Periodic-cm: convolutional recurrent model for crowd density prediction with recurring periodic patterns. In IJCAI, pp. 3732-3738. 2018.
11

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A. THE CUMULATIVE DISTRIBUTION FUNCTION (CDF) OF THE DATASETS
The Cumulative Distribution Function (CDF) of all three datasets is shown in Figure We can find that the TaxiBJ dataset contains more data with larger values, therefore shows different data distribution comparing to the other datasets.
10
0.6
Proportion
Dataset
BikeNYC
TaxiBj
0.4
TaxINYC
0.0
0.0
0.4
0.8
10
Figure 4: The Cumulative Distribution Function of all three datasets.
A.2 VAE LOSS DECOMPOSITION AND TOTAL CORRELATION REGULARISER
Variational Autoencoder (VAE) is generative model that tries to approximate posterior distribution p=(2)>)) by a neural network component qa (z|z). The generative part means that in this model, the data x is considered generated by some random process modelled as po(2)po(z(e) And we optimize 0 to 8* by maximizing the marginal likelihood:
log (x) D (eo(e))))lo + LV A (0));))
Due to the intractability of the posterior distribution, is not likely to calculate the marginal likelihood directly. However, the author provided a variational lower bound (Evidence Lower Bound, ELBO), which can be optimized using stochastic gradient descent:
C(0, xx Dr (xxrllualll))))lll(()))) + Exg( Blo pa p=[=])]]
The second part in Eq. represents the reconstruction loss, and by minimising this term, forces the model to generate reliable reconstructions of the input and the synthesis data with better quality, The first term is a regularisation term which can further tear down for the purpose better disentanglement on the latent representation.
3-VAE (Higgins et al. 2016) is the first model for introducing the disentanglement. By adding a hyper-paramett 3 for penalising the Kullback-Liebler divergence term harder, the representation tends to become disentangled, and the reason behind quite straight forward we decompose that term into the following form (Kim & Mnih 2018):
(gs
T(x, Dr (gs(
The first term describes the nutual information between the input and its latent representation while the second term pushes the distribution of the latent towards the Gaussian prior thus emerging the
12

---

Under review as a conference paper at ICLR 2022
disentanglement. Although the hyperparameter emphasises, it suppresses the model's ability to produce high-quality reconstruction. To address that limitation, the disentanglement term in Eq. is further decomposed into the following form et al. 2018):
Dkrlgo(3)lp(()) = DeL (()))) ((())) - Σ Dkolao(z))ll(z)))
(9)
where IRd and the original ELBO structured as:
-(0,0; = Faa Blg po po(x]|z]]
- I(x,
- DEL
{())
((zg)
Σ DKL (ellae(s)lless
(10)
Not like VAVA,, FactorVAE(Kim & Mnih 2018) and 3-TCVAF ('CCAA((Ceen Chen et al. 2018) introduce the hyperparameter in front of the first term in Eq. 2 which is referred the total correlation (TC) (Chen et al. 2018). By amplifying the penalty on this term, the dependence between the variables is reduced hence emphasising the disentanglement. The second term in Eq. [ is defined dimension- wise KL divergence, which puts constraints on the generated latent code and pushes them towards their predefined Gaussian prior (Li et al. 2020).
The total correlation is intractable since the prior contains mixtures with a large number of components. However, FactorVAF introduces discriminator to approximate the the density ratio that arises in the TC term. In the training session, the model extract representations of two batch of input and then send them to the discriminator. The second batch of extracted representation will be randomly permuted and the goal for the discriminator is to identify whether its input is from g(2) or not. The discriminator and the VAE are trained jointly for getting accurate estimation.
A.3 THE IMPLEMENTATION DETAIL. OF OUR PROPOSED METHOD
Table 3: The detailed structure our proposed method
Parameters
Dataset
Modules
Layer Conv2d
in channel
out channel
kernel size
4
4x4
Imageencoddr
Conv 2d
8
4x4
Conv2d
16
4x4
Conv2d
6
3x3
Feature
Temporal
Separation
6
Gated-Conv
Module
BikeNYC/ TaxiNYC
Conv2d
6
3x3
Conv2d
6
3x3
Conv ConyTranspose2d
16
Decoder
Conv
Transpose2d C
8
4x4
Conv Transpose2d
4
4x4
Linear
256
500
ReLU
Linear
500
500
ReLU
Discriminator
Linear
500
500
ReLU
Linear
500
2
ReLU
A.4 EXAMINING THE LEARNT REPRESENTATIONS
Quantitative evaluation of the learnt representations: We summarise the results in Table As see, approach consistently shows superior performance across all datasets in both the
13

---

Under review as a conference paper at ICLR 2022
disentanglement and reconstruction. This verifies the necessity of extracting time-irrelevan features for each time step, which can capture the unconventional changes in the raster data very well.
Table 4: Quantitative performance comparison on BikeNYC, TaxiNYC and TaxiBJ datasets.
BikeNYC
TaxiNYC
TaxiBJ
MMD
0.858
0.918
0.354
KLD
0.108
0.131
8,305
Original
200-1
1011
200
Ad62
Network Structure
Hotspot
add
odd
6dd 2
add
Spusil
FunturalFuttun Featune
Figure 5: Spatiotemporal data generation controlled by fixing the spatial/temporal variable and modifying the other variable set. The top row shows the synthesis image with changed features, while the images on the second row show the difference between the original image and their corresponding synthetic images.
Generating Attributed Spatio-temporal data: We visualise the spatio-temporal data generated by our method to validate what types of information are extracted by the spatial/temporal features. The generated images are shown in Figure 5 and the original input is placed on the top left corner. The remaining of the top row shows the reconstructed spatiotemporal data with one group features fixed (spatial/temporal) while making changes to the other set of variables. The difference between images on the first row and the original input is calculated and visualised in the second row since those residual images show the impact the reconstructed data when changing the latent representations.
The temporal features, as shown in the right half Figure do not focus on the spatial structure of the data. They capture the temporal influences that flow through the sequence. When are adding negative/positive values to temporal features, the model tends to amplify/weaken the temporal dependencies
can that the areas with non-zero values the previous timestep enhanced/reduced on this timestep. On the contrary, the spatial features contain more than the static structure the transportation network. As shown the left half of Figure when adding/subtracting values to the spatial features, not only the flow of the major roads are amplified/weakenee some hotspot events also emerge across the whole city. The results above demonstrate that verify the necessity of the disentangling design of our approach, which can capture and separate the spatial and temporal features very well.
A.5 REPRESENTATION SWAPPING
Besides the temporal, also perform the representation swapping show the ability of our proposed method to generate synthetic spatiotemporal data. Suppose two real sequences are given for spatial information and temporal information, denoted X, and X,. The synthetic spatiotemporal data expected to preserve the spatial structure in X, and the temporal dependencies in X,. The qualitative comparisons are shown Figure
14

---

Under review as a conference paper at ICLR 2022
Difference
0.0
Figure 6: Spatiotemporal data generation controlled by swapping the spatail/temporal features
15


Reviews:
[{"review_id": "m8DiY_3KOou", "reviewer": "Reviewer_ZJKW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This paper is generally easy to read and follow.\n- The applications considered in this paper are quite relevant and can have a high real-world impact.\n\nWeaknesses:\n- The level of novelty is rather low. This paper builds heavily on ideas from the references (Duan et al., 2019) and (Kim & Mnih, 2018). The authors should make it very clear how this paper differs from the state of the art, and particularly those two references. \n- The motivation for the need for disentangled representations is not clear, nor is there strong empirical evidence suggesting that need. The authors should consider improving the justification of why are disentangled representations necessary for mobility prediction problems. Also, they should consider including empirical evidence of how disentangled representations for spatial and temporal features improve predictive performance - an ablation study could be used for this.\n- The explanation of the methodology is not thorough enough, and a lot of key details are missing from the paper. This hurts readability and reproducibility. For example, how is the sequential prior z_{1:t} = z_t | z_{<t} defined?  In Eq. 6, how is p_\\theta (x_t | z_t^Sp, z_t^Te) defined? How is the total correlation used in the VAE loss defined? \n\"In this work, we estimate the total correlation using the same approach like FactorVAE (Kim & Mnih, 2018), i.e. by introducing a discriminator and using the independence testing trick and the density-ratio trick to approximate the KL term in the above equation.\" -> Can you provide additional details? For example, how is the discriminator defined?\n- The paper does not account, nor refer, to important recent works on modelling spatio-temporal data. Especially, on GNN-based methods, which have recently shown very promising results. A few examples:\n\"Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 753–763, 2020.\"\n\"Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow forecasting. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 364–373, 2021.\"\n\"Yuzhou Chen, Ignacio Segovia-Dominguez, and Yulia R Gel. Z-gcnets: Time zigzags at graph convolutional networks for time series forecasting. International Conference on Machine Learning, 2021.\"\nThe authors should also consider including some of these as baselines. \n- A key baseline is missing: the proposed model without disentanglement. The authors should consider including an ablation study on several key components of their contribution such as total correlation in the VAE loss. This could allow them to justify the need for disentangled representations. \n- The improvements over naive baselines such as a simple ConvLSTM seem quite small. Are these improvements statistically significant? Have the authors considered performing multiple runs and reporting averages, standard deviations and performing statistical significance tests? ", "summary_of_the_paper": "This paper proposes learning a disentangled representation of spatio-temporal mobility data using a VAE-based architecture, which essentially tried to decompose spatial and temporal features and model them independently. Disentanglement in the learnt representation is encouraged through the introduction of total correction as a regularization term in the VAE loss. The authors empirically demonstrate the potential of the proposed approach in 3 different mobility datasets (BikeNYC, TaxiNYC and TaxiBJ). Overall, the results suggest competitive predictive performance with other state-of-the-art approaches.", "summary_of_the_review": "In summary, this paper is interesting to read and easy to follow, but the work needs to be improved before being ready for publication. There are several key issues related to the presentation (e.g. clarity and level of detail of explanations), experimentation, empirical validation of several important claims, state-of-the-art references, and baselines, that should be addressed. Unfortunately, the level of novelty is also rather low.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CgyKjFFFUI4", "reviewer": "Reviewer_MSHW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors approach an interesting and relevant problem such as the one of mobility forecasting. Specifically, the authors focus on how to leverage the disentanglement of spatial and temporal variability in order to get improved forecasting accuracy. I have however some concerns, which I would appreciate if the authors could help me clarify:\n\nIntroduction\n\nThe authors highlight two major challenges in literature: (i) “Difficulty in characterizing dynamic and spatial components”, (ii) “Difficulty in separating the extraction of spatial and temporal features”. Since the authors mention a variety of different works focusing on spatio-temporal forecasting, these claims could benefit from a more concrete narration (e.g. clarifying the meaning of “characterizing the dynamic and spatial components”). The reason being is that this part of the paper is responsible for motivating the additional complexity proposed by the authors. In other words, gaps in the literature, and the ways in which this work aims to fill these gaps, should be clear and tangible. Currently, I feel like I am failing to grasp what the authors mean here.\n\nSection 3\n\n- The authors explain rather quickly the data representation. I feel like a more detailed explanation of how flow data is generated from raw observations of mobility could improve the general readability of this paper. This might be obvious in transportation literature, but not necessarily for a machine-learning audience\n- When defining the priors, the distributional form of $p(z_t^{Te})$ is not given, i.e. what is p(.) in $p(z_t | z_{<t})$?\n- Eq (1):  (i) when defining the $\\prod_{t=1}^T$, it seems like $p(z_t)$ depends on data from the full sequence (1:T), thus also using data from the future. Is this what the authors meant?\n(ii) The authors define $p(z_{1:t}) = \\prod p(z_t | z_{<t})$, with $p(z_t | z_{<t})$ showing a dependency on all previous ts. This is in apparent contrast with Figure 1. If I understand correctly, either (as in Figure 1) the authors assume a Markov property of the latent variables (i.e., z_t summarizes all info up to time t) or the Figure should be adjusted to reflect explicit dependency on all previous timesteps.\nOn the same line, if I am not mistaken, Eq (2) is also not consistent with Eq (1): the role of the product term in the two factorizations is different. In order to be consistent with Eq(1), I guess the authors meant $p(joint) = \\prod_{t=1}^T p(x_t | z_t)p(z_t) = \\prod_{t=1}^T p(x_t | z_t) p(z^{Sp}) p (z^{Te}) = p(x | z) p(z^{Sp}) \\prod_{t=1}^T p(z_t^{Te} | z_{<t}^{Te})$. \n- Eq (3): are the authors actually referring to $z_T$ or should it be $z_{1:T}$? From the factorization used later (product over T of $q(z_t^{Sp} | x_t)$ ) it seems more like the second option.\n- Some background in variational inference could really help for readability (especially how beta-VAEs and FactorVAEs are placed w.r.t. literature, as these seem to be closely related to this work)\n- The authors define the loss as $L_{vae} + L_{tc}$, and say \"we estimate the total correlation using the same approach like FactorVAE\". For clarity of exposition, I would explicitly write how L_tc is computed in the manuscript (authors could consider a background section to group all these key concepts from literature)\n\nExperiments\n- My main concern lies in (i) the misalignment between the experiments conducted and the contributions of the paper, and (ii) the relevance of quantitative results. Authors claim the architecture enables *disentanglement*, but no experiment actually tries to show/validate these claims. Rather, experiments focus on high prediction accuracy. In my view, high accuracy does not prove the better ability of the model to learn disentangled representation, but rather that the specific architecture can achieve better predictive performance\n- Also, the results in Table 1 seem to be somewhat weak. Specifically, authors claim to be SOTA on two out of three, but, if I correctly interpret the Table,  on the TaxiNYC dataset, performance varies depending on which metric is considered (best in MAE, but 5th in RMSE). Also, it can be argued that on TaxiBJ results are not that comparable with the best performing model (having access to means and std. dev over repeated runs would help here)\n- In general, I find the description of the experimental setup could be improved.\n\nMoreover:\n- Is the feature separation model pre-trained and frozen during the training of the other modules in the architecture? Also, the authors say they train the feature separation module on all data, could this have some (unwanted) spill-over of information from future observations? I feel the paper would gain from further elaborating on these points.\n- In 4.2, I am not sure I understand what the disentangle modules are, is it different instantiations of the feature separation module described in section 3?\n- Is the model using only spatial (or temporal) features retrained end-to-end (i.e. is the feature extractor also retrained?)\n- Table 2: are means and std. dev. computed on different hyper-parameter settings? If this were to be the case, I find the statistic to be somewhat uninformative, as the different settings of hyper-parameters represent essentially different models. I would find it more informative if we were averaging over multiple runs of the same configuration. Moreover, if the variance for different hyper-parameters is so high, the authors should elaborate on how to select the best hyper-parameters values.\n- What are the different configurations of hyper-parameters? I don’t seem to find this in the paper.\n- I find the section on informative features very interesting. However, as in the comment above, I don't find the ablation over (unspecified) hyper-parameters very meaningful. For the purpose of exposition, I would probably select the best performing configuration and use that to showcase properties of the model\n\nIn conclusion, here is a list of typos and suggestions that can hopefully be useful:\n- The authors should define terms before using them: ST, ST-raster, \"Closeness, period, trend scheme”\n- End of Section 3.2: \"each frame xt of the by using\"\n- Eq(5) the authors can probably unclutter the equation by aggregating summation terms between the two KL", "summary_of_the_paper": "The authors propose a novel neural architecture to structurally learn disentangled spatio-temporal representations in the context of mobility forecasting. This work argues that predictive models can benefit from enforcing the independence of spatial and temporal dynamics and proposes a VAE-inspired architecture for doing so. \nBy conducting experiments on three mobility datasets, the authors empirically evaluate the prediction performance of the proposed approach against various baselines. Additionally, further experiments attempt to (i) quantify the effectiveness of different temporal features (short-term/daily/weekly correlation), and (ii) formalize and justify a strategy for feature selection.", "summary_of_the_review": "The authors approach an interesting and relevant problem and propose an interesting avenue for improving mobility forecasting techniques. However, I find the main issue of this paper to be the weakness of the empirical results, making the contribution, in my opinion, rather marginal. The methodological section can also be improved, as it leaves many open questions to the reader on a number of design choices. Because of this, I believe the current version of the paper is not yet worthy of publication. \n\nIn my opinion, in order to improve the current version of the manuscript, I believe the authors should focus on:\n- Additional experiments to justify/prove the claims on “spatio-temporal disentanglement”\n- Clarify experimental design (e.g. settings of hyper-parameters)\n- Clarify the significance of results in Table 1\n- Improve/Revise the presentation of the math describing the proposed architecture (i.e., Section 3)\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C93ftWJC40M", "reviewer": "Reviewer_z9DA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. Spatio-temporal representation learning is an important problem.\n2. The proposed solution is evaluated on several real-world datasets.\n\nWeaknesses:\n\n1. The biggest concern for this paper is the motivation. Why is it a good idea to disentangle spatial and temporal representations? This idea does not seem to make sense and is not sufficiently justified in the paper. Typically, the interactions between the spatial and temporal dimensions (i.e., spatio-temporal autocorrelation) is the most challenging component to model in spatio-temporal data. For example, the dynamically changing footprint of traffic congestion on a road network is a function of both location and time. Forcing the separation of the spatial and temporal dimension might cause such dependencies to be ignored in the downstream analysis steps. The authors fail to provide a convincing reason to do so.  \n\n2. It appears that using generative models for spatio-temporal data modeling has been studied previously. The authors may consider checking the following works that use conditional GAN models for traffic estimation:\n\n[1]. Zhang et al. Curb-GAN: Conditional Urban Traffic Estimation through Spatio-Temporal Generative Adversarial Networks. In KDD 2020. \n[2]. Zhang et al. TrafficGAN: Off-Deployment Traffic Estimation with Traffic Generative Adversarial Networks. In ICDM 2019.\n\nWhile VAE is a different generative model, it is very relevant to the above works. The authors may want to demonstrate the differences between their work and these existing methods. They should also be compared in the evaluation section as baselines.", "summary_of_the_paper": "This paper proposes a variation of the Variational Autoencoder (VAE) model to learn disentangled spatial and temporal representations from ST raster data. A separation module is designed in the proposed network. The proposed model is validated on three real-world datasets and achieves better performance compared with the baselines listed in the paper.  ", "summary_of_the_review": "Overall, the paper studies a topic in an important domain. However, the motivation of the work is questionable and unjustified. It does not seem to be beneficial to disentangle spatial and temporal information from an ST dataset. The idea of using generative models for ST data modeling has been explored before, which was ignored by the authors. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "5f1HNsOte9h", "reviewer": "Reviewer_f7db", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* The core idea of the paper, that spatial features are not mandatory static and have to be generated at each timestep is a very interesting research direction in the forecasting setup. \n* The approach is well motivated compared to SOTA  \n* The experiments are well-conducted and supplementary materials give useful insights into the learned representations. \n* The philosophy of the model is well described and the overall architecture is comprehensible.\n\nCons:\n* There are many technical details that are missing to be able to reproduce the architecture and the experiments. The different blocks of the architecture are not detailed (neither in the appendixes), there is no indication for instances on the number of layers, size of the latent spaces, and so on. \n* The description of the model relies too much on references (gated conv. unit, Total correlation are not introduced). \n* A curious aspect of the reported results is not analyzed:  the ablation study shows that the configuration  C0 is clearly the best one, which includes only the closeness without period and trend. What is the point to present the configuration C6 as the main result in this case? And what is the meaning of this phenomenon? The extracted features from the past timestep are sufficient to achieve the best forecasting without taking into account long-range dependencies or seasonality ? It is a very strange result. \n ", "summary_of_the_paper": "The paper proposes a VAE model for mobility forecasting. It focuses mainly on the disentanglement of spatial and temporal features by modelizing explicitly two groups of features in the VAE schema, a group of spatial features that are time-independent and a group of temporal features using a sequential prior. In order to make a prediction, the VAE is used to extract features from three groups of sequences describing the trend, the period, and the last few timesteps before the forecasting period.  An ablation study is provided concerning those three groups of sequences. Experiments show competitive results wrt SOTA.", "summary_of_the_review": "The core idea of the paper is very interesting for the problem of mobility forecasting but as it is the description of the model is too broad to reproduce the architecture and the experiments.  Results show some surprising effects that are not analyzed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed to learn a disentangled representation of spatiotemporal mobility data using a VAE-based architecture, in order to separate spatial and temporal dependencies. This is an interesting and relevant problem, but the reviewers found the paper to be weak in motivation and empirical evaluations.
Decision: Reject