Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposed a novel Gated Inference Network (GIN) to infer and learn state space models in both high and low dimensions. They use concepts in Kalman filtering to obtain estimates of the dynamics. They show that GIN is better at learning state space representations with disentangled dynamics features than existing approaches, they are also robust with noise and able to impute missing data. GIN is efficient in handling the case of unknown dynamics, the models are learned directly from the observations which are fed into encoders to obtain latent observations. 
strength_and_weaknesses: 1. For many case studies, the authors do not estimate the latents but assume the latent form. In a typical dynamics and latent estimation study, we are not provided the latent form (for example, position and velocity in the Michigan NCLT dataset). What happens if the authors want to estimate the latents as well as the dynamics?
2. The paper sets out to perform system identification and learn the parameters of the state space model, but the authors only compare the predicted / estimated states with the ground truth. Certain properties of the recovered matrices can be compared with ground truth, for example, their eigenvalues. The authors need to show that their methods can recover ground truth dynamics.
3. The paper lacks motivation for the architecture of GIN. For example, is the goal of the encoder and decoder simply to reduce dimensionality? What if the model does not have an encoder and decoder, for example in the case of a low-dimensional input (the authors state that they use an MLP in this case, but is this necessary)?
4. The data that the authors used to perform the imputation task is simple although it outperforms other methods shown in the paper, the data has a high amount of temporal correlations, so it may not be hard to achieve an imputation task. Imputation in a complex real-world dataset would be more helpful to show the ability of GIN.
5. The “lack of dynamics” needs to be explained more clearly: it is not clear whether the dynamics are not being inferred or there is no succinct generative dynamical model for the simulations.

Minor comments:
1. The notation of x_{t+} and x_{t-} appear on page 3 first, but their definitions are in the caption of Figure 3 which is later shown.
2. The introduction does not clearly state the problem that the authors are considering. There are many typos and colloquial sentences, e.g., “a bunch of approaches”.

clarity,_quality,_novelty_and_reproducibility: The clarity needs to be much improved. It is not clear which problem the authors are focusing on since they have plugged together a 'high-dimensional, lack of dynamics' study and a 'low-dimensional, with the presence of dynamics' study. There needs to be some structure to understand why these two parts are considered in the same framework.

The ideas in the paper seem promising, but there may be limitations due to the structure of their proposed model (e.g., linear dynamics). These need to be stated clearly.
summary_of_the_review: Due to the lack of clarity and the weaknesses mentioned above, I am not recommending acceptance in the current state of the submission.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes Gated Inference Network, a recurrent architecture for handling noise, missing data, filtering, and smoothing in time series. The methods build on the literature on Kalman Filters and use specific parameterizations for the gain and smoothing matrices representing them as nonlinear transformations of their inputs parameterized by GRUs. Comparisons are done against deterministic architectures such as LSTM and GRU and stochastic models with variational approaches. Both high-dimensional and low-dimensional synthetic datasets are considered for the comparisons showing the scalability of GIN compared to alternatives.
strength_and_weaknesses: **Weaknesses**

**Notation and wording**

Overall, the notation and wording of this paper are confusing and not standard. Please add a notation section and clarify what each variable is representing. More specific comments are below.

* What are latent noisy observations? If a variable is latent it means it's not observed, this wording is confusing for the reader, please fix this.
* What does the superscript + mean? This notation is not common in the statistics literature.
* My current understanding is that $x$ corresponds to dynamics, $w$ corresponds to transformed inputs, and $o$ denotes inputs. Since there are so many letters and new notations used in the paper it would largely benefit from a notation section and will help the reader understand the main contributions better.

**Statistical Grounding**

From a statistical viewpoint, after reading the paper multiple times it is still unclear to me what the generative model and its corresponding graphical model are in this framework. What inference algorithms are used and what approximations are made? More specific comments are below.

* What is the graphical or statistical model?
* Do we start by a model of $x_t|x_{t-1},w_{1:t-1}$?
* If so, does this determine a full joint distribution over all the variables?
* Starting from $x_t|x_{t-1},w_{1:t-1}$ do we first marginalize out $x_{t-1}$ and get $x_t|w_{1:t-1}$ and then to get $x_t|w_{1:t}$ after observing $w_t$?
* What happens if we don't want to use Gaussian distributions? For example, what if the input data is not continuous? Does this framework still hold? What parts do we need to change to adjust for different generative assumptions? If this works only for the Gaussian case this should somehow be reflected in the abstract to clarify the scope of the presented method.
* The Kalman framework seems to rely on linear approximations of the dynamics (where the matrices are given by nonlinear transformations of their inputs). Is the linear approximation a good enough local approximation? 
* The paper seems to lack a solid theoretical or statistical grounding. What type of inference algorithm is used and what is the underlying statistical model? The inference algorithm (if theoretically valid) has to be approximate since the model has nonlinearities and we cannot integrate them out. It looks like the KG and SG matrices are approximated using GRUs for the high-dimensional cases. How do we know if the GRU functions are finding the right KG and SG? What is the justification for using identity functions for low-dimensional cases? How is this reflected in the inference algorithm?


**Presentation**

The presentation and the flow of the paper can be largely improved. I would recommend the authors use the following flow for their presentation.

* First write a summary of Kalman Filtering for completeness to set the context and familiarize the readers with notation (you can add a notation section before this).
* Write a statistical model, draw a graphical model, include parameters, random variables, and noise, and use a better naming convention (latent vs. observed, which variables are following what type of dynamics or equations.
* Write your parameterization (eq. 7-11) and which functions are approximated by GRUs and how this fixed the issue with KF such as matrix inversion, high-d data, covariance parameterization, include a note about what are the approximations to what cost function and how this relates to VI and other approximations.

**Comparisons**

* The method includes lots of matrix multiplications, how does the time complexity and scalability of your method compare to alternatives e.g. VAEs? Can you include a paragraph about the time complexity and report some results on the wall clock time for your methods and comparisons with others?
* The comparison against E2C sounds like a strawman, E2C is an RL algorithm where the representations are learned for a completely different purpose.
* Many other comparisons aren't included both in the introduction and in the compared methods such as SVAE, ARHMM, SLDS,
* For the results reported in tables, did you run all the models until convergence? 
* How is the model complexity accounted for? How does the number of parameters across different models correspond to the number of units?
* You are comparing stochastic and deterministic models. This is not a fair comparison. There are various methods in the VAE literature with specific latent and approximating family structures and algorithms for improving the inference such as importance-weighted autoencoders for tighter bounds, etc. For other models such as ARHMMs and SLDS models, there are specific inference algorithms designed such as Laplace-EM to help with convergence and generalizations. These are not included in the comparisons.
* Test log-likelihoods are not comparable across different models.
* I don't quite understand the sample generation plots, do we want the variance to be small and more densely concentrated around the mean? Although a single parameter has generated the data, we might expect a larger set of parameters to produce similar observations. How should this be quantified properly?

**Minor Issues**
Eq 3, 4: what do you mean by proportional to?
Eq 5, "can be written as a function of $F_t$". This needs to be written down in the main text, it's an important piece of what follows in the manuscript. 
I don't understand eq. 6, it is not a standard notation. Can you clarify this?


## Post Rebuttal

**Presentation** I thank the authors for their detailed responses and clarifications. After reading the author's responses and comments from other reviewers I still think that the paper will largely benefit from a new organization to help clarify it from a statistical viewpoint. Specifically, the graphical models and the notation is not consistent with the literature on state space models (R1, R2, R3, R4) and it makes it hard to identify the contributions and follow the notations. A common strategy for organizing a state space model is to first include a generative process (accompanied by its corresponding graphical model) where there is a clear distinction between random variables, parameters, choice of distributions, functions, etc (R5). Then determine the inference strategy (variational inference, max likelihood, max a posteriori, sampling-based inference, etc.) and determine what approximations are performed. This organization will clearly resolve questions such as the linearity of the model vs. approximation (R7). Including the literature of Kalman filtering as the base notation in the main text will help determine where this method lies in the literature of SSMs (R8). After reading this paper multiple times I still don't have a clear picture of what the generative process is and how this method lies in the statistical models of time series literature.

**Comparisons** My comments and concerns regarding the comparisons did not spark the introduction of new results and comparisons which is against my expectation (R11, R12). In addition, the existing results require more information to be accompanied for completeness such as (effective) model complexity (R13). I also didn't find some of my original questions to be addressed fully such as R15. 

I do believe that the paper poses a really interesting perspective and contribution, and it would be beneficial work to share with the community. Hence I increase my score slightly and encourage the authors to do another iteration of refinement. However, I think the paper is not quite ready for prime time as it is.





clarity,_quality,_novelty_and_reproducibility: I had a hard time following the notation in this paper. The flow of the presentation can largely be improved and the paper can largely benefit from some reorganization. In terms of novelty, deep Kalman filters and variational versions existed before. This paper introduces a specific parameterization in the same context. This specific parameterization did not exist before, but the overall framework is not novel. The results seem to be reproducible although I did not try to reproduce them myself.
summary_of_the_review: The paper could largely benefit from a clear organization and logical flow. Comparisons are not comprehensive and some methods are not considered. The methods presented in the paper lack a solid theoretical and statistical grounding, making it difficult to compare against the existing statistical literature on time series models.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper suggests a state space model to handle both high and low dimensional observation spaces with highly nonlinear observation model, such as animated images of physical systems. The internal state representation is probabilistic, taking account uncertainties. The model is a nonlinear extension of the Hammersein-Wiener model, where the state transition is also allowed to be nonlinear.
strength_and_weaknesses: Strengths:
- Ability to handle high dimensional inputs, such as images directly
- The method builds on existing knowledge from Systems theory, utilizing inductive biases from the field. 
- The experimental setup and the method is very well described
- No costly matrix inversion in the dynamics part

Weaknesses:
- The case of observational uncertainty is solved in an ad-hoc way. While a Kalman filter uses some prior assumption about noise levels, here the un-encoded observations are point observations. It is clear that in the case of high dimensional input this problem is hard to solve better than it is addressed here, for low dimensional cases it may be possible.
- It is not exactly clear how the likelihood without regularization / priors is sufficient to learn a probabilistic representation.

clarity,_quality,_novelty_and_reproducibility: The paper is well written, easy to follow and quite detailed. The experimental setup and the method is very well described (except maybe the loss, assuming there is added regularization terms missing), includes a 24 page long appendix describing the experiments giving pseudocodes, code examples, and the detailed architecture description. This clearly result in good reproducibility.

While I am not perfectly up to date in deep learning extensions of classical Systems theory ideas (this is reflected in my confidence score) the model seems quite novel.

Questions:
What exactly this statement means (in the introduction):
"Moreover, in the variational inference approaches that usually implemented in the context of variational auto encoders
for dimension reduction, they do not have access to the loss directly and have to minimize its lower
bound instead, which reduce the ability of learning dynamics and affect the performance of the model."

While it is true that the exact Bayesian loss (the negative logarithm of the exact posterior over the model parameters) is not available, as it is not available for you either of course, but the expected likelihood of the reconstruction is part of the ELBO with the Kullback-Leibler regularizer. What ability is really reduced in VB relative to your approach? Can you clarify?

(MINOR IMPORTANCE) In the case of low dimensional observation spaces NeuralODE models [1,2,3] can be quite competitive (for example in the case of the Lorenz-system. Do you have a specific reason not to consider these? 
[1] Rubanova, Yulia, Ricky TQ Chen, and David K. Duvenaud. "Latent ordinary differential equations for irregularly-sampled time series." Advances in neural information processing systems 32 (2019).
[2] De Brouwer, Edward, et al. "GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series." Advances in neural information processing systems 32 (2019).
[3] Kidger, Patrick, et al. "Neural controlled differential equations for irregular time series." Advances in Neural Information Processing Systems 33 (2020): 6696-6707.

(MAJOR IMPORTANCE) Maximizing the equation (13) as likelihood seems problematic given choosing high covariance would always give high log likelihood. Do you use any regularization or a prior on the covariance?
summary_of_the_review: The paper is well written and the method seems quite novel.  The claims against Variational Methods should be a bit more clarified. It seems the likelihood itself missing a regularization what a variational objective would have provided. Clarifying what is happening there would be quite illuminating.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a state space model approach to time-series that uses a GRU network to learn the Kalman gain rather than using potentially unstable matrix inversions. 

The numerical results appear convincing, however, the reviewers have reservations mainly with regarding to have the paper is presented. This has prompted the reviewers to propose significant reorganisation of the paper.

The paper definitely merits publication but it is recommended that it is rejected for this conference giving the authors time to properly update the paper for the next.  
justification_for_why_not_higher_score: Getting the paper ready for publication requires substantial updates and therefore also a new round of reviews.
Decision: Reject