Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a VAE model for mobility forecasting. It focuses mainly on the disentanglement of spatial and temporal features by modelizing explicitly two groups of features in the VAE schema, a group of spatial features that are time-independent and a group of temporal features using a sequential prior. In order to make a prediction, the VAE is used to extract features from three groups of sequences describing the trend, the period, and the last few timesteps before the forecasting period.  An ablation study is provided concerning those three groups of sequences. Experiments show competitive results wrt SOTA.
main_review: Pros:
* The core idea of the paper, that spatial features are not mandatory static and have to be generated at each timestep is a very interesting research direction in the forecasting setup. 
* The approach is well motivated compared to SOTA  
* The experiments are well-conducted and supplementary materials give useful insights into the learned representations. 
* The philosophy of the model is well described and the overall architecture is comprehensible.

Cons:
* There are many technical details that are missing to be able to reproduce the architecture and the experiments. The different blocks of the architecture are not detailed (neither in the appendixes), there is no indication for instances on the number of layers, size of the latent spaces, and so on. 
* The description of the model relies too much on references (gated conv. unit, Total correlation are not introduced). 
* A curious aspect of the reported results is not analyzed:  the ablation study shows that the configuration  C0 is clearly the best one, which includes only the closeness without period and trend. What is the point to present the configuration C6 as the main result in this case? And what is the meaning of this phenomenon? The extracted features from the past timestep are sufficient to achieve the best forecasting without taking into account long-range dependencies or seasonality ? It is a very strange result. 
 
summary_of_the_review: The core idea of the paper is very interesting for the problem of mobility forecasting but as it is the description of the model is too broad to reproduce the architecture and the experiments.  Results show some surprising effects that are not analyzed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes learning a disentangled representation of spatio-temporal mobility data using a VAE-based architecture, which essentially tried to decompose spatial and temporal features and model them independently. Disentanglement in the learnt representation is encouraged through the introduction of total correction as a regularization term in the VAE loss. The authors empirically demonstrate the potential of the proposed approach in 3 different mobility datasets (BikeNYC, TaxiNYC and TaxiBJ). Overall, the results suggest competitive predictive performance with other state-of-the-art approaches.
main_review: Strengths:
- This paper is generally easy to read and follow.
- The applications considered in this paper are quite relevant and can have a high real-world impact.

Weaknesses:
- The level of novelty is rather low. This paper builds heavily on ideas from the references (Duan et al., 2019) and (Kim & Mnih, 2018). The authors should make it very clear how this paper differs from the state of the art, and particularly those two references. 
- The motivation for the need for disentangled representations is not clear, nor is there strong empirical evidence suggesting that need. The authors should consider improving the justification of why are disentangled representations necessary for mobility prediction problems. Also, they should consider including empirical evidence of how disentangled representations for spatial and temporal features improve predictive performance - an ablation study could be used for this.
- The explanation of the methodology is not thorough enough, and a lot of key details are missing from the paper. This hurts readability and reproducibility. For example, how is the sequential prior z_{1:t} = z_t | z_{<t} defined?  In Eq. 6, how is p_\theta (x_t | z_t^Sp, z_t^Te) defined? How is the total correlation used in the VAE loss defined? 
"In this work, we estimate the total correlation using the same approach like FactorVAE (Kim & Mnih, 2018), i.e. by introducing a discriminator and using the independence testing trick and the density-ratio trick to approximate the KL term in the above equation." -> Can you provide additional details? For example, how is the discriminator defined?
- The paper does not account, nor refer, to important recent works on modelling spatio-temporal data. Especially, on GNN-based methods, which have recently shown very promising results. A few examples:
"Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 753–763, 2020."
"Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow forecasting. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 364–373, 2021."
"Yuzhou Chen, Ignacio Segovia-Dominguez, and Yulia R Gel. Z-gcnets: Time zigzags at graph convolutional networks for time series forecasting. International Conference on Machine Learning, 2021."
The authors should also consider including some of these as baselines. 
- A key baseline is missing: the proposed model without disentanglement. The authors should consider including an ablation study on several key components of their contribution such as total correlation in the VAE loss. This could allow them to justify the need for disentangled representations. 
- The improvements over naive baselines such as a simple ConvLSTM seem quite small. Are these improvements statistically significant? Have the authors considered performing multiple runs and reporting averages, standard deviations and performing statistical significance tests? 
summary_of_the_review: In summary, this paper is interesting to read and easy to follow, but the work needs to be improved before being ready for publication. There are several key issues related to the presentation (e.g. clarity and level of detail of explanations), experimentation, empirical validation of several important claims, state-of-the-art references, and baselines, that should be addressed. Unfortunately, the level of novelty is also rather low.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a variation of the Variational Autoencoder (VAE) model to learn disentangled spatial and temporal representations from ST raster data. A separation module is designed in the proposed network. The proposed model is validated on three real-world datasets and achieves better performance compared with the baselines listed in the paper.  
main_review: Strengths:

1. Spatio-temporal representation learning is an important problem.
2. The proposed solution is evaluated on several real-world datasets.

Weaknesses:

1. The biggest concern for this paper is the motivation. Why is it a good idea to disentangle spatial and temporal representations? This idea does not seem to make sense and is not sufficiently justified in the paper. Typically, the interactions between the spatial and temporal dimensions (i.e., spatio-temporal autocorrelation) is the most challenging component to model in spatio-temporal data. For example, the dynamically changing footprint of traffic congestion on a road network is a function of both location and time. Forcing the separation of the spatial and temporal dimension might cause such dependencies to be ignored in the downstream analysis steps. The authors fail to provide a convincing reason to do so.  

2. It appears that using generative models for spatio-temporal data modeling has been studied previously. The authors may consider checking the following works that use conditional GAN models for traffic estimation:

[1]. Zhang et al. Curb-GAN: Conditional Urban Traffic Estimation through Spatio-Temporal Generative Adversarial Networks. In KDD 2020. 
[2]. Zhang et al. TrafficGAN: Off-Deployment Traffic Estimation with Traffic Generative Adversarial Networks. In ICDM 2019.

While VAE is a different generative model, it is very relevant to the above works. The authors may want to demonstrate the differences between their work and these existing methods. They should also be compared in the evaluation section as baselines.
summary_of_the_review: Overall, the paper studies a topic in an important domain. However, the motivation of the work is questionable and unjustified. It does not seem to be beneficial to disentangle spatial and temporal information from an ST dataset. The idea of using generative models for ST data modeling has been explored before, which was ignored by the authors.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a novel neural architecture to structurally learn disentangled spatio-temporal representations in the context of mobility forecasting. This work argues that predictive models can benefit from enforcing the independence of spatial and temporal dynamics and proposes a VAE-inspired architecture for doing so. 
By conducting experiments on three mobility datasets, the authors empirically evaluate the prediction performance of the proposed approach against various baselines. Additionally, further experiments attempt to (i) quantify the effectiveness of different temporal features (short-term/daily/weekly correlation), and (ii) formalize and justify a strategy for feature selection.
main_review: The authors approach an interesting and relevant problem such as the one of mobility forecasting. Specifically, the authors focus on how to leverage the disentanglement of spatial and temporal variability in order to get improved forecasting accuracy. I have however some concerns, which I would appreciate if the authors could help me clarify:

Introduction

The authors highlight two major challenges in literature: (i) “Difficulty in characterizing dynamic and spatial components”, (ii) “Difficulty in separating the extraction of spatial and temporal features”. Since the authors mention a variety of different works focusing on spatio-temporal forecasting, these claims could benefit from a more concrete narration (e.g. clarifying the meaning of “characterizing the dynamic and spatial components”). The reason being is that this part of the paper is responsible for motivating the additional complexity proposed by the authors. In other words, gaps in the literature, and the ways in which this work aims to fill these gaps, should be clear and tangible. Currently, I feel like I am failing to grasp what the authors mean here.

Section 3

- The authors explain rather quickly the data representation. I feel like a more detailed explanation of how flow data is generated from raw observations of mobility could improve the general readability of this paper. This might be obvious in transportation literature, but not necessarily for a machine-learning audience
- When defining the priors, the distributional form of $p(z_t^{Te})$ is not given, i.e. what is p(.) in $p(z_t | z_{<t})$?
- Eq (1):  (i) when defining the $\prod_{t=1}^T$, it seems like $p(z_t)$ depends on data from the full sequence (1:T), thus also using data from the future. Is this what the authors meant?
(ii) The authors define $p(z_{1:t}) = \prod p(z_t | z_{<t})$, with $p(z_t | z_{<t})$ showing a dependency on all previous ts. This is in apparent contrast with Figure 1. If I understand correctly, either (as in Figure 1) the authors assume a Markov property of the latent variables (i.e., z_t summarizes all info up to time t) or the Figure should be adjusted to reflect explicit dependency on all previous timesteps.
On the same line, if I am not mistaken, Eq (2) is also not consistent with Eq (1): the role of the product term in the two factorizations is different. In order to be consistent with Eq(1), I guess the authors meant $p(joint) = \prod_{t=1}^T p(x_t | z_t)p(z_t) = \prod_{t=1}^T p(x_t | z_t) p(z^{Sp}) p (z^{Te}) = p(x | z) p(z^{Sp}) \prod_{t=1}^T p(z_t^{Te} | z_{<t}^{Te})$. 
- Eq (3): are the authors actually referring to $z_T$ or should it be $z_{1:T}$? From the factorization used later (product over T of $q(z_t^{Sp} | x_t)$ ) it seems more like the second option.
- Some background in variational inference could really help for readability (especially how beta-VAEs and FactorVAEs are placed w.r.t. literature, as these seem to be closely related to this work)
- The authors define the loss as $L_{vae} + L_{tc}$, and say "we estimate the total correlation using the same approach like FactorVAE". For clarity of exposition, I would explicitly write how L_tc is computed in the manuscript (authors could consider a background section to group all these key concepts from literature)

Experiments
- My main concern lies in (i) the misalignment between the experiments conducted and the contributions of the paper, and (ii) the relevance of quantitative results. Authors claim the architecture enables *disentanglement*, but no experiment actually tries to show/validate these claims. Rather, experiments focus on high prediction accuracy. In my view, high accuracy does not prove the better ability of the model to learn disentangled representation, but rather that the specific architecture can achieve better predictive performance
- Also, the results in Table 1 seem to be somewhat weak. Specifically, authors claim to be SOTA on two out of three, but, if I correctly interpret the Table,  on the TaxiNYC dataset, performance varies depending on which metric is considered (best in MAE, but 5th in RMSE). Also, it can be argued that on TaxiBJ results are not that comparable with the best performing model (having access to means and std. dev over repeated runs would help here)
- In general, I find the description of the experimental setup could be improved.

Moreover:
- Is the feature separation model pre-trained and frozen during the training of the other modules in the architecture? Also, the authors say they train the feature separation module on all data, could this have some (unwanted) spill-over of information from future observations? I feel the paper would gain from further elaborating on these points.
- In 4.2, I am not sure I understand what the disentangle modules are, is it different instantiations of the feature separation module described in section 3?
- Is the model using only spatial (or temporal) features retrained end-to-end (i.e. is the feature extractor also retrained?)
- Table 2: are means and std. dev. computed on different hyper-parameter settings? If this were to be the case, I find the statistic to be somewhat uninformative, as the different settings of hyper-parameters represent essentially different models. I would find it more informative if we were averaging over multiple runs of the same configuration. Moreover, if the variance for different hyper-parameters is so high, the authors should elaborate on how to select the best hyper-parameters values.
- What are the different configurations of hyper-parameters? I don’t seem to find this in the paper.
- I find the section on informative features very interesting. However, as in the comment above, I don't find the ablation over (unspecified) hyper-parameters very meaningful. For the purpose of exposition, I would probably select the best performing configuration and use that to showcase properties of the model

In conclusion, here is a list of typos and suggestions that can hopefully be useful:
- The authors should define terms before using them: ST, ST-raster, "Closeness, period, trend scheme”
- End of Section 3.2: "each frame xt of the by using"
- Eq(5) the authors can probably unclutter the equation by aggregating summation terms between the two KL
summary_of_the_review: The authors approach an interesting and relevant problem and propose an interesting avenue for improving mobility forecasting techniques. However, I find the main issue of this paper to be the weakness of the empirical results, making the contribution, in my opinion, rather marginal. The methodological section can also be improved, as it leaves many open questions to the reader on a number of design choices. Because of this, I believe the current version of the paper is not yet worthy of publication. 

In my opinion, in order to improve the current version of the manuscript, I believe the authors should focus on:
- Additional experiments to justify/prove the claims on “spatio-temporal disentanglement”
- Clarify experimental design (e.g. settings of hyper-parameters)
- Clarify the significance of results in Table 1
- Improve/Revise the presentation of the math describing the proposed architecture (i.e., Section 3)
review_title: null

Meta-review: The paper proposed to learn a disentangled representation of spatiotemporal mobility data using a VAE-based architecture, in order to separate spatial and temporal dependencies. This is an interesting and relevant problem, but the reviewers found the paper to be weak in motivation and empirical evaluations.
Decision: Reject