Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a novel Neural ODE model that embeds the observations into a latent space with dynamics governed by a linear ODE. 
They carefully show that the model satisfies self-consistency, which allows forecasting irregularly sampled time series and have some numerical stability guarantees. They evaluate the performance on medical and climate synthetic datasets, where the model outperforms the similar state of the art models.
strength_and_weaknesses: The theoretical guarantees of the model are nicely derived and clearly presented, and Table 2 is a nice summary of the theoretical advantages of the proposed model. 

The empirical evaluation is compelling, but is on synthetic benchmark datasets only. Including results on a real dataset would make the paper stronger. Even without ground truth, including some visualization of the learnt dynamics vs true dynamics and other models for real datasets would be a great addition to the paper (for example, modeling ocean dynamics)

Figure 1 should also be updated. I think it is of poor quality and doesn't give much insight into the model.

Finally, the OBSERVATIONS section (6.2.) is interesting as it shows the Eigen-values of the kernel matrix, and the authors speculate that the learnt dynamics are periodic. There is space to include some true and inferred dynamics to visualize the periodic signal (or maybe include it in the appendix) 
clarity,_quality,_novelty_and_reproducibility: The paper is very clear and of good quality.
summary_of_the_review: The paper is very clear and shows the the method outperforms previous state of the art models. However, I think the paper would gain in quality with better visualization of the learnt dynamics, in synthetic and real datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors propose to forecast irregular time series using state-space model whose dynamics is specified by a linear ODE. The proposed method meets several desirable properties to time series forecast such as self-consistency, forward stability and allowing for missing values. The experiments show the proposed method achieved competitive performance.
strength_and_weaknesses: Strength:
The continuous-time dynamics, specified by ODE, allows for irregular sampling and missing values.
The linear ODE specification eases the integrator and computational cost.
The linear specification enables Kalman filter
The approach exhibits forward stability and self-consistency.

Weaknesses:
The linear model limits the power of expressive and rules out such as line attractors, limit cycles, strange attractors and etc.
The manuscript seems finished in a hurry. It lacks information.

clarity,_quality,_novelty_and_reproducibility: Clarity:
The full model specification is not in the main text. At least, the distributions of state noise (if any) and measurement(observation) noise are needed. 
More elaboration is needed why the nonlinear Kalman cell is needed for a linear model.
It might be a little confusing or inconsistent using $x$ as ODE variable and then as the measurement (observation).

Quality:
The loss or objective function is not in the main text. It is important as there are trainable parameters.
It is unclear how those parameters are trained, in an offline or online fashion.
The speed of proposed method was not shown in the main text even though claimed fast.
It is unclear why the proposed method performed better than other methods. It's possibly that the dynamics in the datasets happen to be (near) linear, the nonlinear dynamics are harder to train, or by other reasons. It would make the work comprehensive to show the performance with/without model mismatch on synthetic data.
For reader's information and formality, more descriptions are needed in the captions of figures.

summary_of_the_review: This work aims to forecast irregular time series using state-space model whose dynamics is specified by a linear ODE. The proposed method meets several desirable properties to time series forecast such as self-consistency, forward stability and allowing for missing values. 
Several key information seems missing. The writing can be improved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a Neural ODE model that implements a filtering approach for time series. In contrast to previous works, this approach relies on mapping the observations on a linear hidden process. The linearity of the underlying ODE allows fast integration and enforcing several desirable properties such as  consistency and forward stability. The authors show improved forecasting performance on MIMIC and weather forecast datasets. 
strength_and_weaknesses: ## Strengths

- The problem tackled in this paper is very important. Namely, how do we build better latent state space models with irregular and missing data.
- The idea of using a linear hidden ODE is nice as it theoretically can express any complex dynamical system, with a sufficiently large hidden process and expressive observation / embedding function.
- The desiderata and objectives for a desireable time series model are clearly stated 
- The figure 1 regarding the abstraction for a latent space model is very clear and effective

## Weaknesses
### Notation
An important improvement consits in improving the notation. 
- In the problem formulation, it's not clear what $P:\mid t \mid - F$ actually means. 
- Introducing a mask instead of "not-missing" would be appreciated too.
- More importantly, the authors seem to mix $z$ and $x$ over the whole manuscript which makes it unecessarily hard to read. For instance, in Section 4, it seems clear that $z$ stands for the latent process while $x$ is the observation. Now, in section 5, $x$ is now used for the latent space ! (And in Algorithm 2, $z$ is used again). $\hat{x}$ is also sometimes used for the latent space (equatiions 9).

This is a very important point that would greatly benefit the readibility of the paper.

### Lemma 2 

First, it seems that Lemma 2 lacks a proof. What is more, you mention backward stability in D3. How does the matrix exponential fares in terms of differentiability ? I would assume it's not super stable either ?

Also, if your hidden process is forward stable, what does it say about your whole model ? I guess it depends on the output function that you are choosing. So in a sense, you're not ensuring forward stability of your whole state space model.

### Output function

If I understand correctly, your encoding/update function (from observation space to hidden space) is given by Equation 9. What is missing in your manuscript is a discussion of the output function and how it is linked to the update function. 

### Expressivity

One of the important point of that paper is that the desiderata should be achieved without trading for expressivity. In the introduction, the authors hint at the fact that this model can be as expressive as non-linear latent dynamcis models. However, this statement does not get enough focus in the paper. Specifically, I believe an in-depth discussion of the expressivity of the presenteed model should be present in the paper.

### Experiments

- Regarding the non-linear vs linear Kalman cell. It would be nice to have an experiment comparing both approaches.

- The w/hidden version of your model seems to suggest that your latent space and your observation space are the same  (Remark 1)? This seems contradictory with your intro that says that the "observations are nonlinearly mapped into a latent space".

- In general, the experimental section is not very convincing. Indeed, despite the nice layout of the desiderata you need for a good latent state space model, you don't use this blueprint in the experimental section. I believe it would be more impactful to showcase how your model improves upon these metrics. 

- It's not clear what "global existence" mean in Table 2.

- Section 6.2 comes a bit out of the blue and lacks context. So it's hard to appreciate its value in the current state of the paper.

### Minor

- end of page 7, I think "git adLinODEnet" is probably a typo.
clarity,_quality,_novelty_and_reproducibility: ## Clarity

The idea is clear but the notations are inconsistent which makes the overall clarity of the paper low. Additional details are also needed regarding the output and update function of the model.

## Quality

The paper lacks key analysis such as expressivity of the resulting function as well as key experments such as a head to head comparison with GRU-D (for instance on synthetic data).

## Novelty

The paper builds upon a lot of existing work and positions itself very close to GRU-D but with the ability to allows for complex eigenvalues in the state update matrix. The idea explored in itself is novel and will eventually provide a solid paper.

## Reproducibility

The paper is currently not reproducible from the manuscript alone. Indeed, some keys components are still required for full reproducibility (such as the output function). The code has been packaged into a python module.
summary_of_the_review: This paper addresses an important problem but currently fails to provide an in-depth investigation of the proposed approach. The experimental section is not convincing and does not reflect the aims of the paper (the 5 desideratas). The notations are inconsistent, making the overall presentation unclear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a latent ODE model for irregularly sampled timeseries forecasting, where the dynamics in the latent space evolve according to a linear neural ODE. This is enabled by (1) a nonlinear encoder/decoder pair that maps the dynamics between the latent and observed space, and (2) a neural Kalman-like filter operating in the observation space, which updates the states given an observation. By taking the latent space to be higher dimensional than the observation space, the authors justify the use of a linear ODE, reflecting ideas in Koopman theory. The resulting model satisfies a list of desiderata given by the authors and performs well on some benchmark datasets.
strength_and_weaknesses: __Strengths:__
- The proposed model seems to preform quite well on the benchmark datasets, outperforming the state-of-the art in the climate example by a large margin. Though this may simply be due to the higher dimensionality of the latent space used here compared to that in the baselines (perhaps the baselines may perform just as well if they used hidden states of similar size?), the experiments at least demonstrate that the linear dynamics assumption in the latent space is not so limiting.

- The use of a Kalman filter-inspired update rule is nice, enabling some level of interpretability (at least in the linear, shallow case) as opposed to the RNN-type updates used in the other works. The theoretical benefits gained through this choice is also an advantage.

__Weaknesses:__
- Uncertainty quantification does not seem possible within the proposed framework, which may be useful in several applications. Other models such as ODE-VAE and GRU-ODE-Bayes has an advantage in this regard.

- The experiments section feels a little rushed and lacking in quality. For example,
   - One of the reported benefits of LinODEnet is the computational efficiency due to the use of matrix exponentials. A comparison of compute time between the different models would have been desirable to verify this claim. This claimed efficiency is not so clear since there much be some tradeoff between the efficiency gain by linearity vs the need to lift the dynamics to a higher dimension.
   - A novelty of this paper is the introduction of a neural Kalman filter, which distinguishes itself from RNN-style updates. An ablation study to show the efficacy of a self-consistent filter would have been desirable, e.g. by using the same latent dynamical model and encoder/decoder but with different filters.
   - Likewise, some experiments showcasing the performance of the learned filter would have been a great addition, to demonstrate how useful it can be in online settings to perform long-term predictions. For example, including a plot of the filtered trajectory or showing the improvements in e.g. MSE after applying the update step will suffice.

- The writing in some places is not very clear and the presentation can be improved (more details below).

clarity,_quality,_novelty_and_reproducibility: __Clarity + Quality:__
- There is no mention in the body about the "global existence" property highlighted in table 2.
- Some of the notations used in this paper are non-standard and confusing, at least to me:
   - What does $\psi(K)$ mean in Algorithm 2? Is it meant to be $K(\psi)$? i.e. matrix a $K$ parameterised by $\psi$.
   - The star $^*$ notation (for example, $\mathbb{R}^*$) used in section 3 is confusing. Please explain what this means.
- Please clarify which function is required to have model stability (definition 1). Is it just required for the filter component?
- Notation for the filter is inconsistent. For example, in definition 3, a lower case $f$ is used, whereas in Definition 4, an upper case $F$ is used. Note that the vector field of an ODE is also denoted $f$ in this paper, further adding to the confusion.
- The labels in Figure 3 are too small. Please make it bigger.
- What is the `kernel matrix' mentioned in §6.2?
- Some comments on the Appendix:
   - In the proof of self-consistency (section B), please include a line about why the (stacked) filter is idempotent.
   - Section B.1 appears to be incomplete.
   - I am not sure if the argument in section B.2 makes sense. In particular, why do you take $R=\Sigma$? This is generally not true in Kalman filters. I would say the claim "the choice $\alpha=1/2$ corresponds to the Kalman filter" in Lemma 3 is misleading, as this would imply that the two are equivalent in general, although the proof seems to suggest that the correspondence only holds for a very special case of KF.

__Novelty:__
I believe that the ideas introduced in this work are original, in particular the use of linear dynamics in the latent and the Kalman-style update cells, although the technical aspects are not new.

__Reproducibility:__
Sufficient information on the experiments is provided in the appendix for reproducibility, along with dataset and code.

summary_of_the_review: The paper presents a model that, while technically not quite advanced, is original and performs well on several benchmark tasks, proving its capability for use in time series modelling. The paper however suffers from several presentation issues and a weak experiments section.
In light of this, I recommend a light reject, with the possibility of an acceptance provided some of these issues are addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper proposes an approach for time-series forecasting based on encoding observations into a latent space where the dynamics is supposedly linear so that costly numerical integration of general Neural ODE's can be replaced by the matrix exponential. This is paired with a Kalman-like filter component. The paper argues that this construction satisfies a desirable self-consistency property, and can handle missing values, together with some numerical stability guarantees.

Strengths: The problem of learning latent space models for irregular/missing-value time series problems is well motivated and relevant for many applications. The proposal of using a linear latent structure is natural and the method appears to be highly competitive relative to baselines.  The desiderata for a desirable time series model are clearly stated. 

Weaknesses:   The reviewers did not find the experimental section to be entirely convincing in terms of the desideratas setup earlier. The experiments section feels a little rushed and lacking in a comprehensive analysis, e.g. of whether linearity in latent spaces is too restrictive There were also significant presentation issues highlighted, somewhat amplified by inconsistent notation. 
justification_for_why_not_higher_score: Please see weaknesses listed above.
justification_for_why_not_lower_score: N/A.
Decision: Reject