Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a Bayesian, probabilistic approach to data assimilation and parameter estimation for physical systems with unknown observation process. It takes the form of a dynamic VAE integrating in its temporal model the parameterized differential equation that drives the physical phenomenon, allowing the model trained with an ELBO to jointly estimate the unknown parameters and the physical state. The empirical performance of the model is then assessed on three differential equations where its ability to reconstruct the signal and estimate its parameters are demonstrated.
strength_and_weaknesses: ### Strengths

To the best of my knowledge, this paper tackles the **relevant problem** of data assimilation and parameter estimation in the ML x physics community. In this regard, it is **well motivated** as it is designed to operate not directly on the true physical state like many prior approaches, but on an indirect observation of this state whose specifications are unknown. This setting, as well as the designed model, **are novel in the ML x physics domain**. Models and results are, for the most part, **clearly presented and well designed**, with explicit motivations for each of the model components accounting for known and unknown information in the studied phenomenon. **Encouraging experimental results** show the capacity of the model to correctly reconstruct the physical system.

### Weaknesses

There are two weaknesses that limit, to the best of my understanding, the significance of the presented contributions.

Most importantly, **the empirical evaluation of the model does not suffice to properly evaluate the contributions**, because of three main issues.
 - There is a **lack of relevant baselines** in the comparison. Section 5.1 includes no comparison, hence results can hardly be used to draw conclusions. Section 5.2 only considers KVAE, and Section 5.3 only VRNN, GP-VAE and a weak baseline (a non-dynamic VAE). This is insufficient to draw informed conclusions on the relevance of the proposed model. To the very least, VRNN, GP-VAE and KVAE should be included in all experiments. Moreover, other recent baselines could be considered, e.g. Karl et al. (2017, cited in the paper), Krishnan et al. (2017), Li et al. (2018), Yıldız et al. (2019, cited in the paper), de Bézenac et al. (2020), Lu et al. (2020, cited in the paper), to cite probabilistic methods only; note that many ODE/PDE baselines may be considered as well, cf. the Minor Issues section. Note that I am not asking for all these methods to be included but I would expect some of them to be in the paper; alternatively, I would be interested in a discussion from the authors about their relevance in our context.
 - The experiments **fail to prove the relevance of the proposed model w.r.t. the state of the art**. VRNN outperforms the proposed model in reconstruction MSE, and it appears that increasing the dimensionality of KVAE could outperform it as well -- higher values than $n_x = 64$ should be tested. In contrast with the proposed model which requires strong prior knowledge on the physical phenomenon, these baselines are generic and they would be expected to underperform, which is not the case.
 - Even though motivated in Section 3, **main model components are not properly justified with experimental results**. As highlighted in Section 3.1, the introduced dynamics model is stochastic to circumvent uncertainties in the state estimation, but this property is never assessed in the experiments, especially as all considered differential equations are in practice studied in their deterministic version. Moreover, the benefits of adding prior knowledge and inferring the true parameters of the system are not demonstrated in the experiments either: this could be solved by including an ablation study.

Secondly, there are **some clarity issues in the description of the model and its training**, in particular for the non-experts in physical data assimilation like in the ICLR audience. The discretization and implementation of the dynamics model in Section 3.1 are obscure and could benefit from more intuitive explanations. Moreover, the marginalization over $\mathbf{u}$ in Section 4 (paragraph "Nonlinear Filtering") would require further explanations. Finally, **the derivation of the ELBO in Section 4 should be clarified**. Indeed, the proof in Appendix B involves the assumption that the variational posterior is the exact filtering posterior, which is neither discussed nor apparent elsewhere in the paper.

Krishnan et al. Structured Inference Networks for Nonlinear State Space Models. AAAI 2017.\
Li et al. Disentangled Sequential Autoencoder. ICML 2018.\
de Bézenac et al. Normalizing Kalman Filters for Multivariate Time Series Analysis. NeurIPS 2020.

### Minor Issues

 - By design, the proposed model cannot handle non-Markovian data, as the latent dynamics model is a first-order differential equation and the encoder $q_{\phi}$ is factorized as $q_\phi (\mathbf{x} \mid \mathbf{y}) = \prod_n q_\phi (\mathbf{x}_n \mid \mathbf{y}_n)$. This is a limitation of the model that could have been easily avoided like in many deep SSM using filtering or smoothing strategies -- cf. Fraccaro (2018, Section 3.3).
 - The related work misses two relevant lines of work.
   - Some aspects of ML for physics were not addressed. One could refer for examples to the introduction and related work of Yin et al. (2021) containing missing references for parameter estimation and incorporation of physical priors in ML systems.
   - Since the paper deals with video-like input data, it would be beneficial to discuss the relations of the model with existing VAE-based approaches for video modeling, which have already been approached with latent and state-space methods. Cf. for instance the related work of Wu et al. (2021).

Yin et al. Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting. ICLR 2021.\
Wu et al. Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction. CVPR 2021.
clarity,_quality,_novelty_and_reproducibility: ### Clarity

While the paper is **overall well-written**, **some additional clarifications should be made** regarding the description of some components of the model and the derivation of the ELBO (cf. previous section of the review).

### Quality

Besides the aforementioned points, there are a few typos and formatting issues.
 - Section 2 (p. 2): "The Kalman variational autoencoder (KVAE) Fraccaro et al. (2017)" -> The Kalman variational autoencoder (KVAE) of Fraccaro et al. (2017).
 - Section 5 (p. 6): "Our experiment setup is thus mimics" -> Our experiment setup thus mimics
 - Section 3.3 and 4: functions definitions (for $\mu_{\theta}$) should use the `\colon` command rather than `:`.
 - The quality of of all plots in figures should be improved in order to be readable without color information.

### Novelty

Both **the tackled problem and the proposed model** are novel, even though their significance remains unclear given the current experimental results (cf. previous section of the review).

### Reproducibility

While the code is not provided by the authors at submission time, they indicate that they will release it upon publication, which should be checked to ensure proper reproducibility. Besides this, the paper and appendix **contain sufficient implementation details to hopefully replicate the results**.
summary_of_the_review: This paper tackles a well motivated problem and introduces a well designed model to solve it, by integrating physical dynamics prior into a Bayesian, VAE-based model. However, I have concerns regarding the experimental results which I believe are not sufficient to show the relevance of the proposed model w.r.t. prior work, and some clarity issues should be fixed to improve the quality and accessibility of the paper to the ICLR audience. Therefore, I find this paper to lie below the acceptance threshold.

Nonetheless, I believe that my concerns can be addressed during the discussion phase with the authors and the other reviewers, and would be glad to increase my score in this case.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper combines standard VAE with filter algorithms (for example extended Kalman filter). The VAE is used to encode and decode from observation space to the latent space (in the paper often referred to as pseudo-observation space). While the filter algorithm is used to estimate the marginal log likelihood over the latent physics process introduced via additional data likelihood term in the model. The parameters of the model are trained jointly using variational inference (ELBO). The model is tested on video and velocity field datasets. The model obtains similar or better performance compared to previous works in the field w.r.t MSE. 
strength_and_weaknesses: Strengths:
- The paper is very well-written. Most concepts are communicated clearly and the related work section was thorough.
- It touches upon a timely topic, i.e., latent differential equation systems with physical information.

Weaknesses:
- Both the pseudo-observation operator and the noise covariance are assumed to be known in this work. Isn't this too strong of an assumption?
- Since the model and inference heavily resemble KVAE (Fraccaro et al., 2017), the differences should be discussed. In my understanding, the linear dynamical system formulation in KVAE is generalized here. Prior dynamics parameters $\Lambda$ also resemble the coefficients of the linear transition model, i.e., $A$, and $b$ in KVAE; yet, the functional form of the transitions is assumed to be known here (with unknown parameters). Anything else? 
 - Certain claims are not well-supported:
    - The authors mention that they form a ‘physics-informed prior on the latent space.’ However, I would argue that the latent space is the exact physical model rather than just a prior on it. They have argued that the noise term captures model misspecification, however, none of the experiments truly supports this claim: all of your experiments are with the true underlying physical model (with known/partially known or unknown parameters) of the system.
    - The following claim made is not well supported: 'this explicit likelihood is introduced to obtain a well-defined formulation for embedding’. The learned embedding for $x$ is given by the encoder $p(x|y)$. From the formulation, it is not immediately clear how eq7 affects the encoder in order to have ‘well-defined embeddings’. Based on the loss I see that we have a marginal likelihood term for $p(x|\lambda)$ in which we use the data likelihood term defined here. But at the moment I do not see in what way this would affect the variational encoding distribution $q(x|y)$. This should be (at least empirically) demonstrated.
- I have major concerns about the experiments, please see the "quality" point in the next question.

Questions:
- I'm not able to fully follow between eq5 and sec3.2. Is there any novelty here or is it an adaptation of STATFEM into this framework? 


clarity,_quality,_novelty_and_reproducibility: - **Clarity:** As I noted above, the paper is written very clearly! Below are some comments and please see the minor ones at the end of this question.
  - In the last paragraph of the intro, the authors mention that they address the problem of synthesising known physical models with diverse data streams. The formulation is a bit misleading as it can suggest that multiple different data streams can be given as input in the model, which is not the case in the experiments. Hence, I recommend rephrasing. 
  - It would help if the authors would clearly specify in the model design that as the encoding process they see the encoder $p(x|y)$ and then the filtering problem is given by the data likelihood $p(x|u)$ in which they want to marginalise out the latent dynamic variables $u$. Hence, in section 3.2, they actually address the filtering problem. This is not clear from reading this section alone without referring to the supplementary document. 
  - In general, from reading the text it is unclear whether the "latent encodings" refer to the embeddings of the VAE encoder $q(x|y)$ or the data likelihood $p(x|u)$.

- **Quality:** Concerning empirical evaluations, I have significant worries. I would be very happy if the authors can comment on these:
  - Concerning experiments 2 and 3; how meaningful is it to provide comparisons on a dataset generated via the model's own latent dynamics functional form? Don't we naturally expect the presented model to excel?
   - Concerning experiment 1; how realistic is the dataset? Similar approaches typically benchmark their methods on sequence data instead of vector field observations, so I'm not sure if this comparison and presented results translate into real-world scenarios. I would be happy if the authors can clarify the motivation for vector field observations and typical real-world scenarios involving such data.
   - Concerning experiment 2; KVAE assumes unknown pseudo-observation mappings, which would also prevent a fair comparison.
   - Concerning experiment 3; while the dataset comes from a PDE, comparison partners are auto-regressive methods. Is this a fair comparison? I highly encourage comparisons against PDE-based dynamical models.
   -  I am uncertain whether the obtained results support the claim that the rapid optimisation of the ELBO is a sign of the inductive bias forcing the learned representations of the data y to accord with the well-specified latent dynamic model. Indeed the ELBO converges extremely fast, however, MSE suggests that the reconstructions are still improving with an increasing number of iterations, suggesting that either the encoder or decoder or both parameters are still being optimised. As such, I do not see how the fast convergence of the ELBO is a sign that the encoding $q(x|y)$ matches the latent dynamics given by $u$. Similarly, I am missing support for the claim that the final trained model has learnt highly probable latent encodings. 
   - For completeness, I wonder how the model would perform on the datasets that comparison partners are evaluated, e.g., bouncing balls.

- **Novelty:** The paper proposes a combination of existing methods (STATFEM and KVAE). I'm not sure if it the novelty is a strong side of the paper.

- **Reproducibility:** The experimental set-up, parameters and model design are described in the supplementary document. Hence, the paper is reproducible. 


Minor comments:
- In this, **prior belief on** model parameters Λ are updated with data y to give a posterior distribution
- I disagree with the claim "In a typical scenario ... the observation model itself is typically assumed known". The whole literature of GPSSM and the recent latent neural ODE/SDE works do assume unknown observation mappings.
- neural networks are used to learn the unknown embedding **to/from** this lower dimensional space.
- parameters of the likelihood denoted **by**Statistical Finite Elements via Langevin Dynamics.
- "... might be generated only some observed dimensions of ... " is not clear to me.
- I would give eq8 right after eq4 as it is a part of the generative process rather than VI.
- Replacing in the final Gaussian SSM term ‘Observation’ with ‘Pseudo-Observation’ would be better to avoid confusion.
- The specifications of the probability density in sec3.3 seem irrelevant as they are task specific, hence, only needed to be mentioned/clarified in the given experiment set-up.
- For eq9, it would be more clear if the authors write out the true approximate posterior, and then add the claim that they assume that the variational posterior is the exact filtering posterior. 

summary_of_the_review: Overall the paper is very well-written and places itself nicely among the existing work. Acknowledging this is a subjective judgment, I find the proposed approach incremental as (i) many latent ODE/SDE/PDE works are already proposed, and (ii) the model does not achieve groundbreaking results on very interesting datasets. I also think knowing the functional form of the dynamics as well as the pseudo-observation operator and the noise covariance is slightly too restrictive. Certain claims are not well-supported (see "Weaknesses" question). I also have concerns about the datasets, comparison partners. and fairness of the comparisons. I recommend a reject but would be happy to update my review based on the author reply.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a VAE-like framework with a sequential latent variable following a (discretized) PDE. They test it for three different systems.
strength_and_weaknesses: ### Strengths

The paper is basically nicely written and easy to follow.

### Weaknesses

W1.
The relation to the related work is not clearly stated. Given the variety of VAE-like models with sequential structures, it is hard to evaluate the particular design choice of the proposed method unless it is manifested with a clear comparison to the most relevant models.

W2.
Although the authors motivate themselves by referring to the "traditional approaches" that "focus on well-defined observation operators whose functional forms are typically assumed to be known," (in the abstract), in my understanding, the proposed method also stands on an observation operator whose functional form is assumed to be known; the map from $u$ to $x$ is linear (Eq. (7)), and the map from $x$ to $y$ is supposed to be selected with prior knowledge (Section 3.3).

W3.
Experiments are quite artificial and no real data appears.

clarity,_quality,_novelty_and_reproducibility: ### Clarity, quality

The method's description is clear, and the experiments are nicely reported.

### Novelty

As I mentioned in the previous form, the relation with other VAE-like sequential models is not overly clear, which prevents me from assessing the novelty. In my initial understanding, the major difference from previous methods is the use of the discretized nonlinear PDE for state transition. If this is correct, I would say the novelty remains somewhat marginal.


summary_of_the_review: While the paper is nicely written basically, the description of the relation between the proposed methods and other VAE-like sequential models remains unclear. Also, the experiments are only with very artificial data and thus are not so strong.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work combines VAEs and nonlinear state space models to model high dimensional time series data of physical systems.

The proposed $\Phi$-DVAE uses a variational autoencoder to map the high dimensional data in a latent representation (latent observations) whose dynamics can be modeled by a non linear state space model derived from the ODEs or PDEs governing the physical system. Joint estimation of latent states and model parameters is obtained by maximizing the ELBO,  using the extended Kalman filter for state estimation of the SSM.

strength_and_weaknesses: _STRENGHTS_

1. This work in an interesting extension of the  StatFEM framework for modelling high dimensional data derived from an underlying low dimensional physical system with known dynamics.
2. While most of the previous work has focused on modelling simpler dynamics, the proposed model can handle highly nonlinear systems governed by ODEs or PDEs.
3. The added stochasticity of the StatFEM model allows to deal with potential model misspecifications.
4. The model performs well in the chosen experiments, outperforming or performing comparably to competing methods. 

_WEAKNESSES_

1. The StatFEM framework of the dynamic model presented in section 3.1 is quite complex for the average reader in the ICLR community. In its current state i therefore think that this paper would not have the impact it could.
There are a number of improvements that the authors could do to make the paper more accessible, for example:

    1. A simple running example could be used throughout the paper to provide more intuition to the reader.
    2. It would be good to have the full derivation of the dynamic model for the lorenz-63 or advection PDE example (even in the appendix).
    3. You could add pseudo code for the method.

2. In the model you assume the emission distribution of the SSM to be known. How come did you do that? Couldn't you have learnt those parameters as well? 

3. I was surprised to see that the VRNN is slightly better than your model in the Korteweg-de Vries PDE example. Can you provide more intuition on why this is the case? I would have expected a model like the VRNN, which has no inductive bias of physical knowledge, to have a hard time learning to model these compex dynamics.
clarity,_quality,_novelty_and_reproducibility: The usage of the StatFEM within a dynamic VAE is a novel solution that allows to model more complex non-linear systems.

The presentation is clear an well motivated, but needs to be simplified to be more accessible to the average reader (see previous section).
summary_of_the_review: The paper is interesting and novel, but for me to argue for acceptance its exposition needs to be improved and simplified.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper has received mixed reviews, with three reviewers recommending rejection (5-3-5) and one reviewer recommending acceptance (8). The author-reviewer discussion was fruitful and the authors addressed many of the reviewers' concerns.

The reviewers discussed the unclear positioning of the paper during their private discussion. They noted that if the paper's positioning is in the field of probabilistic sequential modelling, then some of the concerns of 66rK and 91wk remain -- and 8UxT acknowledges those. However, if the positioning is in ML x Physics, then relevant baselines and comparisons should be made to assess the paper's contributions better. Regardless of the actual positioning of the paper, the significance and experimental validation of the contribution thus need to be better demonstrated.

Overall, the reviewers recommend a better positioning of the paper as well as an experimental validation that would be more consistent with it. For these reasons, they believe that the paper is not yet ready for publication.
justification_for_why_not_higher_score: The positioning of the paper remains ambiguous, which weakens the experimental validation of the contribution.
Decision: Reject