Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a general formulation of SSMs, called the Non-Parametric State-Space Model (NPSSM). This framework is able to depict the non-stationarity of the latent process over time. Based on the identifiability of time-lagged latent variables, they proposed a new structural VAE for model estimation and forecasting tasks. They optimize the reconstruction of the observation and the prediction of the latent from the latent at the previous time. 
strength_and_weaknesses: 1. It is not clear how the ‘time-varying change factors’ c_t enter into the transition equations. Specifically, they are not present in the latent predictor model, nor in the loss function (apart from the KL). It is unclear if they are being used at all, and if so, how.
2. The metrics of comparison for the true and estimated latent variables can be much improved. The authors can learn an optimal permutation to compare the latent variables.
3. The invertibility of the transmission plays an important role in the proposed model: the model seems to be under an assumption that the transition is invertible. Please discuss the situation where the transition and change transition functions are not invertible. This would seemingly lead to errors in the inference.

Minor
1. In Figure 4, the readers are not able to know the order of the latents in scatter plots, in particular, which one is the latent 2 in the MCC matrix?
2. Some notations in equations need clarifying, e.g., ‘i’ and ‘j’ in Eq 2 need to be clearer, the difference between z_t and \hat z_t in Eq 7 should be clarified, the notations in A.3.2, and so on.
3. Grammar: page2, paragraph 2, line 3,  ‘, In particular’.

clarity,_quality,_novelty_and_reproducibility: The introduction and the motivation of the work are clear. The novelty of the work is unclear since it is difficult to ascertain the utility of the change factors.
summary_of_the_review: The work is interesting, but the clarity of the methods needs work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a SSM adopting functional causal model for transition and post nonlinear model for emission, inferred in a VAE framework. The method was motivated to overcome several limitations of existing SSM formulations including the use of additive noises, and handling non-stationary time series. Experiments were conducted on synthetic and real-world datasets with a small number of selected baselines.
strength_and_weaknesses: Strengths:


The use of functional causal modeling in SSM is interesting and promising. The issues of dealing with non-additive noises and non-stationary transition functions are also important.



Weakness:

The writing of the papers is generally difficult to follow. It was not clear how some of the stated goals were achieved. 

- The paper emphasizes "non-parametric" models, referring presumably the use of casual function modeling for the transition process. How is this model effectually reflected in the estimated SSM is quite unclear. As described in page 5-6, the latent prediction model p_pred of z essentially is modeled as a LSTM, and the noise is generated from the inverse latent transition function that is modeled as MLPs. It is not clear where is the latent causal model in the final ELBO loss in Equation (7). Is it in p(z_1:T) and p(c_1:T)? If yes, what are their relations to the "latent prediction model"? Why do we need both the latent causal model (equation 2) and latent prediction models at the same time? All of these questions made it difficult to assess the contribution of the presented causal transition functions.

- How is the 2nd term in the ELBO loss (7) supervised at training time? What's the significance of this term?

- The baselines used are limited. As revised in the related works, there is a large set of SSMs with complicated modeling of q(z) and p(z)'s, such as those below. These should be included in the comparisons in order to truly understand the contribution of the presented work over state of the arts.

Krishnan et al, Deep Kalman Filters, 2015

Klushyn et al, Latent matters: Learning deep state-space models, NeurIPS 2021


- Please elaborate on the definition and calculation of MCC in the main text. In the Appendix, it is stated that 'It first computes the absolute values of the Spearman’s rank
correlation coefficients between every ground-truth factor against every estimated latent factor". It is not clear how is this done in the temporal setting. What are ground-truth factors in the synthetic experiments and ho are the "estimated latent factor" determined? How are all the permutation done in a time series setting for all z_t's over time? How are these done for the baseline models?

- Given that the method introduced two rather complicated components to state-of-the-art SSMs, an ablation study is needed to understand the importance of each of the components (causal model, and non-additive noises)


- As the authors discuss identifiability, it seems that it mainly refers to the identifiability of the latent variables z_t's. Does the identifiability of z_t's directly translates to the identifiability of the underlying transition function? In traditional SSM, the identifiability of the system states and system parameters (that define the transition functions and others) are not necessarily the same?
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper can be improved. In the present form, some of the critical technical pieces are unclear, especially regarding the role of the causal model in the SSM (see details above). As a result of this, it was difficult to truly assess the novelty of the presented method and the reproducibility also becomes limited.

summary_of_the_review: This paper discusses two interesting ideas for improving deep SSMs. However, the lack of clarity, the insufficient consideration of comparison models, and the lack of ablation study made it difficult to assess the actual impact of these two ideas on improving SSM-based time-series modeling and forecasting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposed a new State Space Model (SSM) for forecasting. The proposed work relies on 1) non-parametric functional causal model (Pearl, 2009) for the transition process and 2) post-nonlinear model (Zhang & Hyvarinen, 2012) to capture nonlinear distortion effects in the emission model. The resulting model is named as NPSSM. They also extend it with time-varying change factors, which serves as input to the transition process, to model the non-stationary aspect (N-NPSSM).

The authors proved, under certain conditions, their proposed NPSSM is identifiable. The proof extends the theorem in Yao et al., 2022 to additive noise case with post non-linear function (their formulation of NPSSM). An estimation framework based on structural VAE is proposed and experiments on 4 synthetic datasets and 3 real-world datasets demonstrated superior performance over 4 deep-forecasting and deep state-space models.

strength_and_weaknesses: The strength of the paper is the balanced contributions in theoretical, methodological and empirical aspects. It is a small step towards a more general and flexible SSM for forecasting tasks.

The weakness of the paper mostly is the experiment. Even though it seems strong empirically, the setting is not very convincing for me. Below is a list of comments on possible way to improve experiments.
* There are no results for NPSSM and include it may serve a good ablation study. 
* The author mentioned many deep learning and deep state space models in the related work but 4 are chosen including a naive LSTM. Including more baselines for sure could help.
* Nowadays, 3 datasets are very limited from my perspective. 
* Looking one step ahead is fine but would be better if multi-step ahead is also considered. 
* If using one step ahead for evaluation, one simple baseline can be the last seen value.
* Why R_{0.9} loss instead of considering the average of all the quantiles? 
* The hyperparameters and training detail are missing for the baselines.
* Regarding to the MCC in Table 1: Is it fair to compare latent space with other algorithms when the data generation largely follows the proposed architecture?

clarity,_quality,_novelty_and_reproducibility: The clarity is fine, but can be improved. There are several typos and I list a few that I noted while reading. 
* In Sec 4: “...variables are sparse an…” an -> and
* Paragraph before Sec 5.1, “We compare NPSSM with typical deep forecasting models
”. But in Table 1 and 2, N-NPSSM is used.

The quality of the work can be improved in both theoretical and empirical sides, especially on the experimental results as already mentioned in the weakness of the paper. On the theoretical side, I only have a minor comment. The proposed theorem is for NPSSM, but the main results in Table 1 and 2 are all for N-NPSSM (time varying version of NPSSM). Fixing this mismatch would strengthen the usefulness and impact of the theorem. 

I think the novelty is also fine, it is a small step towards a more general and flexible SSM for forecasting tasks.

The reproducibility is fine for me, training and architecture details are provided in the appendix, but not the code.
summary_of_the_review: The paper has balanced contributions in theoretical, methodological and empirical aspects. It is a small step towards a more general and flexible SSM for forecasting tasks. But the theoretical and methodological contributions are extensions of previous works, limiting the significance. More importantly, the experiments section need to be improved so that the empirical results are more convincing. Please refer to Strength and Weakness section for more detail.

=== After authors' response ===
The authors addressed my questions on the empirical side and I increased my score to 6.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a state-space model for time series modeling inspired by the general structural causal models. 

Strength:
- The method does theoretically make sense and tries to propose a solution to an important problem. 
- The theoretical results after the discussion period and clarifications are correct and useful in understanding the proposed method.

Weaknesses: 
- Unfortunately, the paper misses connecting their work to the most important developments from the sequence modeling community which is Structural State-Space models which share so much with the work of authors. There is a full body of related works that must be discussed in this paper both theoretically and experimentally. Here is a non-exhaustive list of works on SSMs as a powerful sequence modeling tool that has recently gotten popular:

[1] Gu, A., Goel, K., & Ré, C. (2021). Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396

[2] Gupta, A. (2022). Diagonal State Spaces are as Effective as Structured State Spaces. arXiv preprint arXiv:2203.14343.

[3] Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., & Ré, C. (2021). Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34, 572-585.

[4] Hasani, R., Lechner, M., Amini, A., Liebenwein, L., Tschaikowski, M., Teschl, G., & Rus, D. (2021). Closed-form continuous-depth models. arXiv preprint arXiv:2106.13898.

[5] Gu, A., Dao, T., Ermon, S., Rudra, A., & Ré, C. (2020). Hippo: Recurrent memory with optimal polynomial projections. Advances in Neural Information Processing Systems, 33, 1474-1487.

[6] Smith, J. T., Warrington, A., & Linderman, S. W. (2022). Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933.

[7] Hasani, R., Lechner, M., Wang, T.H., Chahine, M., Amini, A. and Rus, D., 2022. Liquid structural state-space models. arXiv preprint arXiv:2209.12951.

[8] Gu, A., Gupta, A., Goel, K. and Ré, C., 2022. On the parameterization and initialization of diagonal state space models. arXiv preprint arXiv:2206.11893.

[9] Gu, A., Johnson, I., Timalsina, A., Rudra, A. and Ré, C., 2022. How to train your hippo: State space models with generalized orthogonal basis projections. arXiv preprint arXiv:2206.12037

These works not only are the SOTA in sequence and time-series modeling but also directly relate to this paper. Including these frameworks in the paper would require a major revision and is a necessary condition for the acceptance of the paper. 


- The authors violated the anonymity condition when providing their code. The identity of the authors is revealed in the git logs, unfortunately.
Decision: Reject