Spatio-temporal Disentangled representation learning for mobility prediction

1 INTRODUCTION . Spatio-temporal prediction tasks like mobility forecasting are critically important for smart city applications ( Zheng et al. , 2014 ) . With the help of the rapid deployment of IoT ( Internet of Things ) devices and sensors , massive amounts of saturated ST datasets are available , and many researchers have put their efforts to improve the performance of ST prediction ( Zhang et al. , 2016 ; Wang et al. , 2018 ; Zonoozi et al. , 2018 ; Jin et al. , 2018 ; Yao et al. , 2019b ) . A central problem in deep learning for crowd flow prediction tasks is the extraction of features from both spatial and temporal domains . Researchers first tried to extract the entangled spatio-temporal features directly from the data ( Zhang et al. , 2016 ) . Then , they tried to extract features from spatial and temporal domains separately . Wang et al . ( 2018 ) and Zonoozi et al . ( 2018 ) proposed models that extract periodic representations or short-term temporal features directly from the ST data using recurrent-based convolution operations and find their effectiveness in producing more accurate predictions . Based on that , in STDN ( Yao et al. , 2019b ) , the explicit extraction of the long-term periodic information was shown to also improve the prediction . Although the existing approaches appear to be powerful in terms of results and predictions , two major challenges hinder these models : 1 ) Difficulty in characterising dynamic and spatial components . Spatio-temporal correlation is more complex since it comprises dependencies from both the spatial and temporal regions . Models like LDRSN ( Tian et al. , 2020 ) and RegionTrans ( Wang et al. , 2018 ) are capturing temporal dependencies explicitly . However , their complicated structure makes it hard to characterise and validate the effectiveness of their dynamic and spatial components . 2 ) Difficulty in separating the extraction of spatial and temporal features . Except for the spatial and temporal features from the most recent data , many approaches try to extract long-term/periodical features to improve the performance . However , without an explicit separation mechanism , the extracted long-term temporal dependencies introduce irrelevant noises to the predictor ( factors do not vary with time and are only relevant to the long-term sequence ) . The extent to how generative models can model , extract and disentangle the spatial and temporal features in ST-raster data is an open problem . To address the above challenges , we introduce the disentangled representation learning to the mobility forecasting task . Disentangled representation learning , which separates the learnt representation into independent variables such that each variable relates to one semantic factor of sensory data ( Bengio et al. , 2013 ) , offers a solution for the problem mentioned above . For a spatio-temporal task like mobility forecasting , an ideal disentangled representation should have the ability to separate time-relevant components from the factors that don ’ t vary with time , which can help improve the predictor ’ s performance . Many prior works have explored disentangled representation learning for spatio-temporal data ( Hsu et al. , 2017 ; Li & Mandt , 2018 ; Denton & Birodkar , 2017 ; Zhu et al. , 2020 ) . Most of them assume that the features can be disentangled into two sets : a set of dynamic features that extract the temporal correlations and a set of static features that describe factors that are constant through the input sequence . However , since most of them focus on the movement of some predefined objects , their assumption is inaccurate when applying to datasets for mobility forecasting . For example , a sudden car accident might show no relationship to the temporal influence from the previous timestep and the static traffic network structure . Therefore , there are still gaps in how to model the spatio-temporal data using the disentangled representation learning method . To address this problem , we assume that each timestep has its own temporal features and features that do not vary with time . By doing so , our model can better formulate the spatio-temporal data , and we can achieve controlled data generation frame by frame . In this work , we proposed a VAE-based model to learn disentangled spatio-temporal representation . Compared to the conventional methods for the ST prediction problem , our approach extracts the entangled features first and then explicitly separates them into temporal variables and spatial variables using disentangled representation learning method . This will force the model to keep the learnt spatial/temporal features as mutually exclusive as possible . For applying disentangled representation learning method to real ST data , we assume that each frame has its own spatial and temporal variables and separate these two groups with auxiliary regularisation . It helps the model to formulate the complicated spatio-temporal sequence . Our experimental results ( see Section 4 ) show that the learnt representations have a similar level of performance with the current state-of-the-art methods . Our key contributions can be summarised as follows : 1 . We propose a novel approach to learn disentangled spatio-temporal representations for mobility forecasting tasks . The learnt representation is separated into two independent groups : spatial and temporal factors . 2 . We conducted several experiments on multiple spatio-temporal datasets and used the learnt representation for mobility forecasting . Results show that our methods achieve state-of-theart performance compared to other baseline mobility forecasting methods . 3 . We investigate the effectiveness of our methods under the ” Closeness , Period , Trend ” scheme and how to further improves the model ’ s performance by selecting the informative features from the learnt representations . 2 RELATED WORK . Deep spatial-temporal networks for mobility forecasting : In order to make accurate traffic prediction , many researchers have paid attention to capturing the spatio-temporal dependencies hidden behind the traffic data . Besides the conventional methods like Seasonal ARIMA ( Moreira-Matias et al. , 2013 ) , deep learning methods are increasingly used in more and more works for mobility forecasting . ST-ResNet ( Zhang et al. , 2016 ) was proposed to capture the spatial dependencies through a stack of residual convolution layers . It also stacks the frame from a near and a distant time period to capture temporal dynamics . The goal of using the residual units is to overcome the gradient vanishing problem , and the results show that capturing distant spatial features can improve the performance of mobility forecasting . DeepSTN+ ( Lin et al. , 2019 ) proposed a ResPlus unit that is capable of capturing long-range spatial correlations . Although they succeed in extracting distant spatial features , they lack attention on capturing the temporal dependencies . To further explore the effectiveness of temporal features , recurrent-based approaches were introduced to capture the temporal correlations . PCRN ( Zonoozi et al. , 2018 ) was proposed , which first extract entangled spatio-temporal representation using a convolutional recurrent network ( CRN ) and then updating the periodic representations by CRN ’ s hidden state . Attention-based LSTM methods are adopted by STDN ( Yao et al. , 2019b ) and ST-DCCNAL ( Li et al. , 2019 ) , which try to capture long-term temporal dependencies . In summary , our proposed method differs from other methods in the explicit separation of spatial and temporal features using a disentangled representation mechanism . Disentangled Representation Learning for Spatio-temporal data : Most prior works on the disentangled representation learning problem are developed based on Variational Autoencoders ( VAE ) ( Kingma & Welling , 2013 ) , which is an unsupervised generative learning method . β-VAE , proposed by Higgins et al . ( 2016 ) , forces the inference model to disentangle the latent representation by adding a new hyperparameter β to create an information bottleneck on the prior . FactorVAE ( Kim & Mnih , 2018 ) further breaking down the objective function and try to enhance disentanglement by penalising the total correlation of the learnt representation . As for sequence modelling , a number of prior publications have extended VAE to video and speech data ( Fabius & Van Amersfoort , 2014 ; Chung et al. , 2015 ; Bayer & Osendorfer , 2014 ) . These models , although being able to generate realistic sequences , do not explicitly disentangle the representation of time-invariant and time-dependent information . Thus it is inconvenient for these models to perform tasks such as controlled generation . S3VAE ( Zhu et al. , 2020 ) is proposed to separate static and dynamic factors of sequential data . Another approach proposed by ( Li & Mandt , 2018 ) is also focusing on separating the dynamic factors from static factors . Although they share a similar idea which uses an RNN-based architecture to extract dynamic factors for each timestep , they use different prior setups . Each frame in ( Li & Mandt , 2018 ) has its own content features while the time-invariant variables are shared by the whole sequence in S3VAE . As for the spatial-temporal type of data like video , SV2P ( Babaeizadeh et al. , 2017 ) uses the variational model to extract the time-invariant latent and make predictions for multiple frames . Models like ( Denton & Birodkar , 2017 ; Hsieh et al. , 2018 ) try to factorise each frame into a stationary part and a temporally dynamic component . 3 SPATIO-TEMPORAL VAE MODEL . 3.1 SPATIO-TEMPORAL DATA . In this work , we will useD = { Xi } i=1 : N to denote a spatio-temporal raster dataset that comprises N i.i.d . sequences . Each X ≡ x1 : T = { x1 , x2 , ... , xT } in that dataset denotes a sequence of raster data with T frames . Since we focus on mobility forecasting tasks using grid-based spatial representations , which all have the similar dimensionsH×W . Hence each frame xt ∈ RH×W represents the mobility flow of a certain area at a given time interval t . 3.2 SEPARATING SPATIAL FEATURES FROM TEMPORAL FEATURES . Many prior works have explored disentangled representation learning for spatio-temporal data . Models like FHVAE ( Hsu et al. , 2017 ) tried to separate global variables from segment ( dynamic ) variables for speech data . DSVAE ( Li & Mandt , 2018 ) , DRNET ( Denton & Birodkar , 2017 ) and S3VAE ( Zhu et al. , 2020 ) aim at the disentangled representation for video , which also factorised latent variables into static and dynamic parts . However , current approaches for learning the disentangled representation on sequence or spatio-temporal data like video assume that there is a fixed content or object shared by all frames in the sequence since they are using video datasets like the Stochastic Moving MNIST ( Denton & Fergus , 2018 ) and Sprite ( Li & Mandt , 2018 ) . Sequences in these datasets often comprise images describing the movement of a set of the same numbers or virtual avatars . Under this context , static features describing this object can be extracted , and its performance can be evaluated by the accuracy of a classification task . However , for most of the real spatio-temporal datasets , the static features are not enough . For example , in a traffic flow dataset , in addition to the fixed structure of the traffic network and the temporal influences , there might be some hotspot suddenly emerges . At that specific time , the rise of those events shows no correlation to the temporal features and the fixed content . Therefore , we assume that for each frame , data is generated based on two sets of features : a set of temporal features which comprise the influence from the sequence before it and a set of spatial features that describe the structure of the network and local events . In this work , we propose a novel architecture that extracts mixed ( entangled ) feature maps for each timestep in the input sequence and then separates them into temporal and spatial variables . It allows us to analyse and controlled generate each frame separately . The architecture of our method for spatio-temporal disentangled representation learning is illustrated in Fig.1a . It consists of three major modules : ( i ) Image Encoder to yield initial feature maps from the input spatio-temporal sequence . ( ii ) A Feature Separation module to separate the spatial and temporal variables from the initially mixed feature maps . ( iii ) A Decoder to reconstruct the sequence frame by frame based on their corresponding spatial and temporal variables . The Image Encoder comprises several layers of CNN , and the feature maps are extracted from each frame separately . The goal here is to yield some higher-level features for the next step . An Instance Normalisation layer is used after each convolutional layer to improve the reconstruction results since using batch normalisation might remove the instance-specific contrast information from the data , which is useful in the later step ( Ulyanov et al. , 2016 ) . Intuitively , without any disentanglement constraints , the feature maps that come out of the Image Encoder should be the entanglement of features from both spatial and temporal domain . Therefore , the goal for the Feature Separation module is to separate them explicitly . For temporal features , inspired by ( Yu et al. , 2017 ; Gehring et al. , 2017 ) , we use a Gated Convolutional structure to capture temporal dynamics from the mixed feature maps . This temporal gated convolution layer contains two 1-D causal convolution layers . As Fig.1b shows , those 1-D convolution operations will be applied on the time axis for each pixel of the feature maps , which means the capture of the temporal dynamics from each mixed high-level feature . It is much easier to understand the usage of the Image Encoder here since the 1-D convolutional operation on the high-level features map is definitely better for separating the temporal dynamics than applying them on the individual pixel of the original input . The Decoder module is used to reconstruct each frame xt by using their corresponding spatial variables zSpt and temporal variables z Te 1 : t as input .

Summary: The authors propose a novel neural architecture to structurally learn disentangled spatio-temporal representations in the context of mobility forecasting. This work argues that predictive models can benefit from enforcing the independence of spatial and temporal dynamics and proposes a VAE-inspired architecture for doing so.  By conducting experiments on three mobility datasets, the authors empirically evaluate the prediction performance of the proposed approach against various baselines. Additionally, further experiments attempt to (i) quantify the effectiveness of different temporal features (short-term/daily/weekly correlation), and (ii) formalize and justify a strategy for feature selection.