Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed LS4 a sequential latent state space model with the latent state evolving according to a discretized approximation of ODE. The structure of the latent state dynamics induces an efficient convolutional implementation. The model is trained in a VAE framework. Experiment results on challenge datasets including data from stiff systems show strong performance of the model
strength_and_weaknesses: Strength:

1. The motivation of introducing a latent variable for S4 model is strong. The model leverages the strong representation power of latent variables and efficient convolutional implementation of S4 Model.
2. The work compares against a broad set of baseline models. The experiment results show strong performance and efficiency of the proposed model.


Weakness:
1. Some important baselines are missing from the comparison including VRNN[1] and latent SDE[2].
2. For efficient convolutional implementation, the dynamics of the latent variable are limited to linear dynamics. It’s not clear if such transition dynamics are going to limit the representation power of the latent state. That’s also why I think models with more flexible transition dynamics like VRNN and latent SDE should be compared.
3. The work reports ELBO in experiments but it is not a tight bound of log-likelihood. IWAE as well as its particle filtering version for sequential latent variable model FIVO are tighter than ELBO and can better reflect how well the generative model fits the data.
4. The technical novelty of the proposed model is incremental. The model can be viewed as a special case of the existing sequential latent variable model or latent extension of the S4 model. This is a minor concern as the efficiency improvement based on convolution could compensate for the lack of novelty.


[1] Chung, Junyoung, et al. "A recurrent latent variable model for sequential data." Advances in neural information processing systems 28 (2015).

[2] Li, Xuechen, et al. "Scalable gradients for stochastic differential equations." International Conference on Artificial Intelligence and Statistics. PMLR, 2020.
clarity,_quality,_novelty_and_reproducibility: I have no big concerns about the clarity or writing quality of the paper. I do have concerns about the work’s novelty. Please refer to Strength and Weakeness for more details.
summary_of_the_review: I appreciate the extension of S4 models to its latent variant which leverages both the representation power of sequential latent variable models and the efficiency of S4 models. But I'm especially concerned about the lack of comparison against models with more flexible latent variable models like VRNN and the recently proposed latent SDE.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes LS4 which is a generative model for sequences inspired by the deep state space model (SSM) S4. LS4 performs latent space evolution following a state space ODE and is trained via sequence VAE objectives. State-of-the-art results on selected datasets for continuous-time latent generative models are reported.
strength_and_weaknesses: 
 - [+] The paper is well written and easy to follow. All necessary theoretical and conceptual background information is concisely introduced.
 - [+] The reviewer also appreciates that comprehensive source code is added in the supplementary materials.
 - [-] In several passages of the text “significant” performance gains are claimed although no statistical tests are performed. Furthermore, there is no indication that several replicates per method were
       performed and there are no error bars in the respective result tables.
 - [-] The authors do not explain how the hyperparameters for the respective methods were selected. In section D only one specific hyperparameter setting of LS4 is explained: how was this selected and what
       is the exact hyperparameter search space of LS4 and the baseline methods?
 - [-] It is not explained how 4 of 30 datasets were selected in section 5.2. Why weren’t datasets from baseline methods like SaShiMi or SDEGAN selected?
 - [-] Since SaShiMi is specifically designed for modeling raw audio waveforms, wouldn’t S4 or a variant thereof have been a more suitable baseline? This is also briefly discussed by the authors in the last
       paragraph of D.1.
 - [-] The information in Figure 1 is not clearly illustrated, and the main caption is missing. The plot in (a) is quite overloaded and the y-axis label should be more specific (e.g. “value of x”). It is not
       clear how well LS4 approximates the ground truth. Is it possible to plot the mean trajectories with standard deviations for a fixed p? Some more information on the FLAME problem would be helpful. How
       do different trajectories arise given a fixed p, does the variation come from varying initial conditions?

### Questions

 - What are the runtimes of SaShiMi and other VAE-based methods in Table 1? Can the authors explain why the runtimes of the baseline methods experience exponential growth, shouldn’t this be on scale
   O(N^2 L)? Why do the runtimes of Latent ODE drop by increasing the sequence length from 80 to 320? It would also be interesting to compare memory consumption in an experiment since shorter runtime can
   come at the cost of more overhead and the practical implementation does not necessarily have to follow the theoretical result in Proposition 4.2.
clarity,_quality,_novelty_and_reproducibility: The theoretical part of the paper is clearly written but the experimental section needs clarification in order to evaluate the quality of the results.
The paper is novel in a way that it combines the S4 approach for SSM with the VAE framework for data generation.
summary_of_the_review: The paper is well written and easy to follow. Nevertheless, important information needs to be added in the description of the experimental process to be able to correctly assess the presented results.

### Minor Questions

 - Why wasn’t the FID used for comparing the generated with the real distribution in section 5.2? It is a well-established metric for generative models and e.g. the bottleneck layer of an Autoencoder could
   have been used to compute the Frechet Distance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an extension of S4 layers [Gu et al. 2021] with latent variables, which are used to define a generative model. 
To this end, the authors define an auto-regressive prior distribution over latent variables leveraging the S4 parametrisation. Furthermore, the approximate posterior is defined as a product of marginals, and also leverages S4 parametrisation. 

strength_and_weaknesses: While the paper is generally well written, clarification is needed for several details:

For instance, the model and posterior before Eq. (5) makes assumptions that need explanation: 
- The emission model is both auto-regressive in all observations x and all latent variables z. Can you motivate why you consider the dependence structure on x additionally to z? And why use auto-regressive instead Markovian models which are more common in SSM literature?
- The inference model assumes that the (approximate) posterior factorises as a product of marginals, which is very restrictive. Why is this choice made as a starting point?
- Eq. (5) appears to be in conflict with the equation for the approximate posterior above. Whereas above, you assume that the approximate posterior factorises as a product, here you use a factorisation into conditionals (which is not restrictive). Which factorisation did you actually consider? And is Eq. (5) the objective you use in the end? It's not obvious as that is in the background section.
- In Eq (5), the posterior is conditioned on observation only until some intermediate x_ti, but it should be the full trajectory of observations, otherwise you have an additional gap in the ELBO. This has been discussed e.g. in [1]. (Unless you would do a filtering-based approximation for all the conditionals p(y_t | y_<t), but that seems also not the case.)

Another thing that confused me is that in the beginning x is used as inputs and y as target observations. And later, x are the observations of the generative model. I think it would be good to either use a different symbol for one, or clarify this better.

I don’t understand Eq. (8). x_tk and z_tk are scalars, and h_tk is the feature vector at time-step t. How does the convolution operation apply here? Maybe you want to write the convolution for the sequence, similar to Eq. (3)?

What is H_\beta? In the inline text above its mentioned that it is the function mapping x, z to h. What is that function? Only dh/dt has been specified, but not h itself, or how it is approximated. Bilinear transformation?
Later, there is H_\beta_1 and other subscripts. This is undefined. What is the index?

In 4.1, before starting the paragraph "LS4 prior block", I am missing a connecting sentence. It took me a while to understand that the block parametrises the conditional distribution. Because the next equation has "y" as output and there is no other equation for z or mu_z and sigma_z, this paragraph read as unconnected to that equation. This can be streamlined better for the reader.  

I am also a bit worried and uncertain in my understanding about how a latent trajectory is generated. For the posterior, as far as I understand, you can compute the approximate posterior marginals in parallel, because you factorise it as a product of marginals? If so, this should be noted. For the prior, can you only compute the hiddens h and y in parallel, and then have to sample auto-regressively? Is this the reason why there are no forecasting experiments? If this is a limitation, it would be great to state this explicitly. 

Why is the prior on z discrete-time, whereas h is continuous-time? I think this is never mentioned. 

Eq (11), what is G_yz?  suppose its just an additional matrix. Why this additional transformation here?

“In turn, state–space models do not suffer from stiffness of dynamics as the numerical methods are sidestepped in favor of an exact evaluation of the convolution operator.”
I would like to learn a bit more about this. As far as I understand, the convolution operator is applied to the *discretized* system. How does this sidestep numerical problems? 

How do you deal with initialisation? From what I understand from the previous work, the Hippo initialisation is one of the key things for the empirical performance. I think this was not discussed, despite its importance. And there is also no theory regarding how to adjust this for the additional latent variables z. 

Related work.
I think the paper is missing a quite large literature of related work on deep state-space models that are not of the S4 type. Among the vast literature on deep variational state space models, several methods even use conditional linear or linearised state space models, for instance [2,3,4,5].


Experimental evaluation.
The performed experiments show that the model is able to generate realistic samples even for quite challenging data distributions. This seems quite promising. However, I find the choice of experiments and baselines not ideal. In particular, 
- why not compare to the obvious baseline S4? Both in terms of speed and performance.
- since the model uses latent variables to express uncertainty, I would have expected to see some experiments and evaluation metrics that would benefit from this, e.g. probabilistic forecasting evaluated with the CRPS score.
The current set of experiments leave me somewhat undecided. On the one hand, the model seems to be able to generate realistic data very well. I am not sure if this implies that the latent transition model is also learned properly, or whether there is potentially a short-cut, e.g. using only the initial latent state and ignoring the dynamics. This speculation would be easily debunked through a forecasting experiment. In the experiments in 5.3, the worse performance on Physionet is explained as if the randomness in the latent space was a limitation, but that should be actually the selling point of this model, as certain data just is not perfectly predictable. 


[1] Bayer et al. 2021, Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models
[2] Rangapuram et al. 2018, Deep State Space Models for Time Series Forecasting
[3] Fraccaro et al., 2017, A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning
[4] Kurle et al. 2020, Deep Rao-Blackwellised Particle Filters for Time Series Forecasting
[5] Klushyn et al. 2021, Latent Matters: Learning Deep State-Space Models
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written in terms of language. But I think there are several gaps that need to be filled. Reading the paper now a few times still leaves me confused about several details. Examples are given in Strengths and Weaknesses. I think these need to be improved, but many of them can be addressed easily.

I think the paper is not particularly novel, as it is a direct extension of previous work (S4), combining it with variational inference. 
This combination did not require novel technical contributions. On the other hand, this is the first paper proposing a probabilistic extension of S4. 

In the current form, I would not be able to reproduce the paper, but I hope to gain more clarity from the rebuttal/revision.
summary_of_the_review: I think this paper has great potential and it is overall good work. However, in the current version, I find several things unclear, and would therefore vote for rejection. Many questions and clarification could be addressed in the rebuttal though.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The papers discusses a latent state-space model that evolves according to an approximate discretised ODE. This allows for an efficient implementation using convolutions and training via VAE.  The main challenge the reviewers faced was with the clarity of the paper, with some technical issues in the original submission and concerns that the work appears somewhat incremental. Whilst these may have been addressed to some extent, the overall reviewer sentiment remained that the paper isn't yet strong enough to merit acceptance.
justification_for_why_not_higher_score: The reviews for this paper were consistently low and the authors were not able to upgrade their scores sufficiently after rebuttal.
Decision: Reject