Spatio-temporal Disentangled representation learning for mobility prediction

1 INTRODUCTION . Spatio-temporal prediction tasks like mobility forecasting are critically important for smart city applications ( Zheng et al. , 2014 ) . With the help of the rapid deployment of IoT ( Internet of Things ) devices and sensors , massive amounts of saturated ST datasets are available , and many researchers have put their efforts to improve the performance of ST prediction ( Zhang et al. , 2016 ; Wang et al. , 2018 ; Zonoozi et al. , 2018 ; Jin et al. , 2018 ; Yao et al. , 2019b ) . A central problem in deep learning for crowd flow prediction tasks is the extraction of features from both spatial and temporal domains . Researchers first tried to extract the entangled spatio-temporal features directly from the data ( Zhang et al. , 2016 ) . Then , they tried to extract features from spatial and temporal domains separately . Wang et al . ( 2018 ) and Zonoozi et al . ( 2018 ) proposed models that extract periodic representations or short-term temporal features directly from the ST data using recurrent-based convolution operations and find their effectiveness in producing more accurate predictions . Based on that , in STDN ( Yao et al. , 2019b ) , the explicit extraction of the long-term periodic information was shown to also improve the prediction . Although the existing approaches appear to be powerful in terms of results and predictions , two major challenges hinder these models : 1 ) Difficulty in characterising dynamic and spatial components . Spatio-temporal correlation is more complex since it comprises dependencies from both the spatial and temporal regions . Models like LDRSN ( Tian et al. , 2020 ) and RegionTrans ( Wang et al. , 2018 ) are capturing temporal dependencies explicitly . However , their complicated structure makes it hard to characterise and validate the effectiveness of their dynamic and spatial components . 2 ) Difficulty in separating the extraction of spatial and temporal features . Except for the spatial and temporal features from the most recent data , many approaches try to extract long-term/periodical features to improve the performance . However , without an explicit separation mechanism , the extracted long-term temporal dependencies introduce irrelevant noises to the predictor ( factors do not vary with time and are only relevant to the long-term sequence ) . The extent to how generative models can model , extract and disentangle the spatial and temporal features in ST-raster data is an open problem . To address the above challenges , we introduce the disentangled representation learning to the mobility forecasting task . Disentangled representation learning , which separates the learnt representation into independent variables such that each variable relates to one semantic factor of sensory data ( Bengio et al. , 2013 ) , offers a solution for the problem mentioned above . For a spatio-temporal task like mobility forecasting , an ideal disentangled representation should have the ability to separate time-relevant components from the factors that don ’ t vary with time , which can help improve the predictor ’ s performance . Many prior works have explored disentangled representation learning for spatio-temporal data ( Hsu et al. , 2017 ; Li & Mandt , 2018 ; Denton & Birodkar , 2017 ; Zhu et al. , 2020 ) . Most of them assume that the features can be disentangled into two sets : a set of dynamic features that extract the temporal correlations and a set of static features that describe factors that are constant through the input sequence . However , since most of them focus on the movement of some predefined objects , their assumption is inaccurate when applying to datasets for mobility forecasting . For example , a sudden car accident might show no relationship to the temporal influence from the previous timestep and the static traffic network structure . Therefore , there are still gaps in how to model the spatio-temporal data using the disentangled representation learning method . To address this problem , we assume that each timestep has its own temporal features and features that do not vary with time . By doing so , our model can better formulate the spatio-temporal data , and we can achieve controlled data generation frame by frame . In this work , we proposed a VAE-based model to learn disentangled spatio-temporal representation . Compared to the conventional methods for the ST prediction problem , our approach extracts the entangled features first and then explicitly separates them into temporal variables and spatial variables using disentangled representation learning method . This will force the model to keep the learnt spatial/temporal features as mutually exclusive as possible . For applying disentangled representation learning method to real ST data , we assume that each frame has its own spatial and temporal variables and separate these two groups with auxiliary regularisation . It helps the model to formulate the complicated spatio-temporal sequence . Our experimental results ( see Section 4 ) show that the learnt representations have a similar level of performance with the current state-of-the-art methods . Our key contributions can be summarised as follows : 1 . We propose a novel approach to learn disentangled spatio-temporal representations for mobility forecasting tasks . The learnt representation is separated into two independent groups : spatial and temporal factors . 2 . We conducted several experiments on multiple spatio-temporal datasets and used the learnt representation for mobility forecasting . Results show that our methods achieve state-of-theart performance compared to other baseline mobility forecasting methods . 3 . We investigate the effectiveness of our methods under the ” Closeness , Period , Trend ” scheme and how to further improves the model ’ s performance by selecting the informative features from the learnt representations . 2 RELATED WORK . Deep spatial-temporal networks for mobility forecasting : In order to make accurate traffic prediction , many researchers have paid attention to capturing the spatio-temporal dependencies hidden behind the traffic data . Besides the conventional methods like Seasonal ARIMA ( Moreira-Matias et al. , 2013 ) , deep learning methods are increasingly used in more and more works for mobility forecasting . ST-ResNet ( Zhang et al. , 2016 ) was proposed to capture the spatial dependencies through a stack of residual convolution layers . It also stacks the frame from a near and a distant time period to capture temporal dynamics . The goal of using the residual units is to overcome the gradient vanishing problem , and the results show that capturing distant spatial features can improve the performance of mobility forecasting . DeepSTN+ ( Lin et al. , 2019 ) proposed a ResPlus unit that is capable of capturing long-range spatial correlations . Although they succeed in extracting distant spatial features , they lack attention on capturing the temporal dependencies . To further explore the effectiveness of temporal features , recurrent-based approaches were introduced to capture the temporal correlations . PCRN ( Zonoozi et al. , 2018 ) was proposed , which first extract entangled spatio-temporal representation using a convolutional recurrent network ( CRN ) and then updating the periodic representations by CRN ’ s hidden state . Attention-based LSTM methods are adopted by STDN ( Yao et al. , 2019b ) and ST-DCCNAL ( Li et al. , 2019 ) , which try to capture long-term temporal dependencies . In summary , our proposed method differs from other methods in the explicit separation of spatial and temporal features using a disentangled representation mechanism . Disentangled Representation Learning for Spatio-temporal data : Most prior works on the disentangled representation learning problem are developed based on Variational Autoencoders ( VAE ) ( Kingma & Welling , 2013 ) , which is an unsupervised generative learning method . β-VAE , proposed by Higgins et al . ( 2016 ) , forces the inference model to disentangle the latent representation by adding a new hyperparameter β to create an information bottleneck on the prior . FactorVAE ( Kim & Mnih , 2018 ) further breaking down the objective function and try to enhance disentanglement by penalising the total correlation of the learnt representation . As for sequence modelling , a number of prior publications have extended VAE to video and speech data ( Fabius & Van Amersfoort , 2014 ; Chung et al. , 2015 ; Bayer & Osendorfer , 2014 ) . These models , although being able to generate realistic sequences , do not explicitly disentangle the representation of time-invariant and time-dependent information . Thus it is inconvenient for these models to perform tasks such as controlled generation . S3VAE ( Zhu et al. , 2020 ) is proposed to separate static and dynamic factors of sequential data . Another approach proposed by ( Li & Mandt , 2018 ) is also focusing on separating the dynamic factors from static factors . Although they share a similar idea which uses an RNN-based architecture to extract dynamic factors for each timestep , they use different prior setups . Each frame in ( Li & Mandt , 2018 ) has its own content features while the time-invariant variables are shared by the whole sequence in S3VAE . As for the spatial-temporal type of data like video , SV2P ( Babaeizadeh et al. , 2017 ) uses the variational model to extract the time-invariant latent and make predictions for multiple frames . Models like ( Denton & Birodkar , 2017 ; Hsieh et al. , 2018 ) try to factorise each frame into a stationary part and a temporally dynamic component . 3 SPATIO-TEMPORAL VAE MODEL . 3.1 SPATIO-TEMPORAL DATA . In this work , we will useD = { Xi } i=1 : N to denote a spatio-temporal raster dataset that comprises N i.i.d . sequences . Each X ≡ x1 : T = { x1 , x2 , ... , xT } in that dataset denotes a sequence of raster data with T frames . Since we focus on mobility forecasting tasks using grid-based spatial representations , which all have the similar dimensionsH×W . Hence each frame xt ∈ RH×W represents the mobility flow of a certain area at a given time interval t . 3.2 SEPARATING SPATIAL FEATURES FROM TEMPORAL FEATURES . Many prior works have explored disentangled representation learning for spatio-temporal data . Models like FHVAE ( Hsu et al. , 2017 ) tried to separate global variables from segment ( dynamic ) variables for speech data . DSVAE ( Li & Mandt , 2018 ) , DRNET ( Denton & Birodkar , 2017 ) and S3VAE ( Zhu et al. , 2020 ) aim at the disentangled representation for video , which also factorised latent variables into static and dynamic parts . However , current approaches for learning the disentangled representation on sequence or spatio-temporal data like video assume that there is a fixed content or object shared by all frames in the sequence since they are using video datasets like the Stochastic Moving MNIST ( Denton & Fergus , 2018 ) and Sprite ( Li & Mandt , 2018 ) . Sequences in these datasets often comprise images describing the movement of a set of the same numbers or virtual avatars . Under this context , static features describing this object can be extracted , and its performance can be evaluated by the accuracy of a classification task . However , for most of the real spatio-temporal datasets , the static features are not enough . For example , in a traffic flow dataset , in addition to the fixed structure of the traffic network and the temporal influences , there might be some hotspot suddenly emerges . At that specific time , the rise of those events shows no correlation to the temporal features and the fixed content . Therefore , we assume that for each frame , data is generated based on two sets of features : a set of temporal features which comprise the influence from the sequence before it and a set of spatial features that describe the structure of the network and local events . In this work , we propose a novel architecture that extracts mixed ( entangled ) feature maps for each timestep in the input sequence and then separates them into temporal and spatial variables . It allows us to analyse and controlled generate each frame separately . The architecture of our method for spatio-temporal disentangled representation learning is illustrated in Fig.1a . It consists of three major modules : ( i ) Image Encoder to yield initial feature maps from the input spatio-temporal sequence . ( ii ) A Feature Separation module to separate the spatial and temporal variables from the initially mixed feature maps . ( iii ) A Decoder to reconstruct the sequence frame by frame based on their corresponding spatial and temporal variables . The Image Encoder comprises several layers of CNN , and the feature maps are extracted from each frame separately . The goal here is to yield some higher-level features for the next step . An Instance Normalisation layer is used after each convolutional layer to improve the reconstruction results since using batch normalisation might remove the instance-specific contrast information from the data , which is useful in the later step ( Ulyanov et al. , 2016 ) . Intuitively , without any disentanglement constraints , the feature maps that come out of the Image Encoder should be the entanglement of features from both spatial and temporal domain . Therefore , the goal for the Feature Separation module is to separate them explicitly . For temporal features , inspired by ( Yu et al. , 2017 ; Gehring et al. , 2017 ) , we use a Gated Convolutional structure to capture temporal dynamics from the mixed feature maps . This temporal gated convolution layer contains two 1-D causal convolution layers . As Fig.1b shows , those 1-D convolution operations will be applied on the time axis for each pixel of the feature maps , which means the capture of the temporal dynamics from each mixed high-level feature . It is much easier to understand the usage of the Image Encoder here since the 1-D convolutional operation on the high-level features map is definitely better for separating the temporal dynamics than applying them on the individual pixel of the original input . The Decoder module is used to reconstruct each frame xt by using their corresponding spatial variables zSpt and temporal variables z Te 1 : t as input .

Summary: This paper proposes learning a disentangled representation of spatio-temporal mobility data using a VAE-based architecture, which essentially tried to decompose spatial and temporal features and model them independently. Disentanglement in the learnt representation is encouraged through the introduction of total correction as a regularization term in the VAE loss. The authors empirically demonstrate the potential of the proposed approach in 3 different mobility datasets (BikeNYC, TaxiNYC and TaxiBJ). Overall, the results suggest competitive predictive performance with other state-of-the-art approaches.