Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a continuous latent time series model called the Continuous Latent Process Flow (CLPF). Irregular time series observations are modeled as invertible transformations (i.e. normalizing flows) of an underlying stochastic process (the base process), where the flow transformation is itself parameterized by a time-varying stochastic process (the latent process), modeled by a (learned) stochastic differential equation (SDE). The authors also develop a piecewise variational approximation to the latent process, which they train by maximizing the importance-weighted ELBO. In contrast to prior work like Continuous Time Flow Processes (CTFP) and latent SDES, the authors show that CLPFs better model synthetic and real-world time series, including processes which require non-Lipschitz transforms of base processes, and non-Markovian processes. CLPFs also better capture qualitative trends in real world data.
main_review: This was a well written paper that by-and-large demonstrated its quality beyond existing methods, both empirically and mathematically. While the clarity of the figures and exposition of the math could use a bit of improvement, the experiments are thorough, and are well-designed to illustrate the differences between CLPFs and prior work, making the work of significance to the field. I especially appreciated the inclusion of ablation studies to show the importance of each component. The use of an SDE to parameterize a flow-based transform is also original, to my knowledge, as is the piece-wise variational approximation. In light of the originality, quality and significance of the proposed method, I recommend acceptance of the paper.  Greater clarity would have improved my score to an 8. I just have some clarificatory questions, and some suggestions to improve how the model and its math are communicated.

My main clarificatory question is why it's important to have _both_ a base process (which is transformed to the observed process) and a latent process (which parameterizes the transformation) in order to achieve modeling flexibility. I understand how this choice improves over CTFPs, which do not handle non-Lipschitz transformations explicitly. But what about something like the latent SDE with suitable augmentations: 
1. Model the base process using an SDE (instead of the OU or Wiener process)
2. Transform that using a CTFP-style continuously-indexed flow to the observable process
3. Use a piecewise SDE variational approximation so that the posterior can be non-Markovian

Would the worry about non-Lipschitz transforms apply to this? (And is this actually one of the ablated baselines tested? I couldn't tell). Because this would avoid the need to have two unobserved stochastic processes, which seems like it would reduce computational cost. Basically, I think it'd be great to better explain why you need both the $O_t$ process and the $Z_t$ process, instead of just transforming the $Z_t$ process to the $X_t$ process directly. (The other alternative is transforming the $O_t$ process to the $X_t$ process directly, which I understand would be a variant of CTFP.)

Apart from that, I wanted to share a few points that were confusing to me about the figures and the math. For Figure 1, it was not immediately obvious to me, without reading the paper closely, that the latent process wasn't the same as the base process. It's also generally not clear to me what is illustrated on the right-hand-side of the picture. Are the grey lines showing (stochastic) transformations) of specific values of the base process to the observed process? Is the blue line showing the evolution of the observed process, or something else? I think it'd be helpful to show samples of both the base process and the observed process (like in the CTFP) paper, and then on top of that show how the latent process parameterizes transformations from one sample of the base process to one sample of the observed process.

For Figure 2, the graphical model was confusing to me, because it omits / marginalizes out the base process $O_t$. Intuitively, I was thinking of a CLPFs as a HMM-style graphical model with *two* hidden states at each $t$, $O_t$ and $Z_t$, and so I was surprised to see the non-Markovian dependence of $X_{t_i}$ on $Z_{t_{i-1}}$. It was only after I thought a while about the math shown on Eq. 9 that I realized that marginalizing out $O_t$ leads to the dependence of $X_{t_i}$ on $Z_{t_{i-1}}$, but this wasn't clearly shown / derived. I think it would be a good idea to revise both the Equations and Figure 2 to better explain how this dependence arises.

Finally, I think it might be a good idea to call the reweighting term $M^{(i)}$ in Eq. 14 an importance weight, since that is a readily recognized term for reweighting samples from a proposal distribution (in this case, the approximate posterior).

== Post-Rebuttal ==

Thank you to the authors for their detailed responses, clarifications, and additional experiments. I continue to believe this is significant paper that makes meaningful methodological contributions to the field, and I think this has been further demonstrated by the additional experiments. I also appreciate that the author's are aware of the method's limitations and social impact, and look forward to seeing their inclusion in the final paper. I will maintain my score of 7.
limitations_and_societal_impact: I think this paper could do with more discussion of limitations, especially relative to existing methods. Besides the issues with numerical approximation that were (very briefly) noted, one potential limitation that comes to mind is that CLPFs have a higher training, likelihood evaluation, and inference cost, due to the absence of a tractable likelihood. This is in contrast to CTFPs, which can take advantage of a deterministic transformation between the base and observed process, and do not require variational inference for training. Presumably this means that CLPFs might require more time / compute / energy train to reach the same performance on some tasks, and to perform likelihood evaluations, which might motivate using other models (even if they are less flexible).

The authors did not discuss societal impacts. I would encourage them to do so -- perhaps there are positive/negative impacts of being able to model real-world continuous time data more faithfully, e.g. for forecasting and scientific modeling -- but I think it is their prerogative, given that this is more of a "general methodology" paper rather than an application-focused paper.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper introduces CLPF---novel parametrization of a latent SDE---that has higher expressivity than previous approaches. CLPF is obtained as a time-dependent invertible transformation of a base process.
This transformation is implemented as a normalizing flow whose parameters are a function of another SDF.
In addition, the authors introduce a novel non-Markov piecewise variational posterior process that elegantly circumvents the fact that SDE solutions are Markov. The method is evaluated on four well-designed syntethic datasets, whose purpose is to test different properties of CLPF. The efficacy to model more complicated data is tested on a real-world dataset, while an ablation study evaluates the relative importance of different model components.
main_review: Originality: 
- This work combines several existing approaches and algorithmic improvements. Namely, CLPF extends Continuous-Time Flow Process (CTFP). CTFP is defined as an invertible transformation of a base process. This work changes the base process from Wiener to OU, which prevents the variance of the process from growing over time. It also introduces a time-varying flow, whose parameters are a function of another (latent) SDE. Finally, the authors introduce a novel piecewise variational posterior process which improved learning of the generative model and inference. 
- It is clear how this work differs from the predecessors, which are adequately cited (I might have missed some relevant works as I'm not an expert in neural SDEs).

Quality:

- I am not familiar with SDE-based ELBOs, so I cannot comment on the correctness of the derivations.

The good:
- This paper seems to be well done. 
- It is well written. The method is clearly explained. 
- The experiments are well designed---I especially like the synthetic data experiments, where each experiment has a clear purpose (as it tests a different aspect of the model).
- The piecewise posterior approximation is novel and it is cleverly designed to circumvent a known limitation.

The bad:
- I think that in Eq 15, when going from line 1 to line 2, the authors missed the expectations with respect to previous latent variables. This is a common thing to not include them in sequential VAEs. However, this is incorrect if not stated explicitly. 
- The authors do not discuss any limitations of the method. I wonder what they are, if any?
- In experiments, the VRNN (a discrete-time baseline) often outperforms continuous methods in terms of the marginal likelihood. I find that weird. While the authors mention that VRNN is unable to produce samples consistent with the likelihood, I do not understand why that is, and it is not explained.
- Fig. 3 contains qualitative evaluation but compares the proposed method only with a weak discrete-time baseline. Please include a better continuous-time alternative.

Questions:
- NODEs are notoriously slow and expensive to compute. CLPFs include three different neural SDEs (base for the likelihood, posterior and a latent one). How does that affect computational cost with respect to a vanilla NODE? How about with respect to VRNN (used as a baseline)?
- In a discrete-time system, a state could be engineered such that a Markov system becomes a non-Markov due to inclusion of additional information in the state. While I understand that your piecewise posterior idea solves the issue, I wonder if a similar solution to a discrete-time process could also be employed?
- Have you thought about using a deterministic neural ODE as an encoder instead of a discrete-time RNN? I think this could gracefully handle non-uniform temporal sampling. This could be formulated as integrating some hidden state between observation time-stamps, and discontinuous state updates (conditioned on an observation) at time-stamps with available observations.

Clarity:

- The paper is generally well-written and contains a sufficient amount of information about the method. It does not contain any details about the neural nets used or optimisation. The authors mention that the details are deferred to the appendix, but I haven't checked.
- It would be nice to have a few comments on the interpretation of certain quantities, e.g., what does M_T in Eq. 2 correspond to? Can I interpret M^{(i)} in Eq. 14 as an importance weight? Was does the u_2 - u_1 term in L82 mean?
- How do I interpret Eq. 3? As a non-expert in ODEs, I do not understand why that form is limiting.
- Eq 11 and 13 could be easier to read with a change of variables e.g. \hat{t_i} = t_{i_1} + \tilde{t} or similar.

Significance:
- The problems studied in this paper are currently very popular within the community.
- This work is quite original and significantly improves on prior art by improving benchmark results as well as meaningfully expanding the methodological toolkit.
- In my opinion this is a very significant piece of work.


limitations_and_societal_impact: Authors do not discuss any limitations of this work (e.g. computational or memory cost, where it would fail compared to alternatives). A societal-impact discussion is absent but also not necessary for this paper.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper addresses probabilistic modeling of sparse or irregularly sampled time series. The proposed method captures the systems dynamics by a rather simple latent stochastic process, which is then warped into a more complex one using a time-indexed ANODE-like normalizing flow. The free parameters of the dynamical system are identified by maximizing the ELBO that lower bounds the likelihood of the observed sequence. The solution aims to overcome the limited expressive power of normalizing flows stemming for the necessity to use invertible transformations that also need to satisfy Lipschitz continuity restrictions.
main_review: Significance of Novelty
--

A key design choice is to assume the latent dynamics to be governed by an Ornstein Uhlenbeck process, which is a rather trivial stochastic process with analytical solution. The beauty of the solution comes when this simple process is warped by a normalizing flow into dynamical models with arbitrary complexity. This is a compelling solution, and however simple, a novel use of stochastic differential equations in modern machine learning. The suggested variational inference algorithm, especially piecewise reweighting part is also interesting and novel for the context in question. However, there do exist alternative solutions to all the problems pointed out in the paper, which appear to work in the reported experiments better than the proposed method. The lack of empirical evidence on the benefit of the approach impairs its significance.

Presentation
--
Section 2 gives a really nice, concice, and clear overview into the essential material to be used later. It is remarkable how a variety of complex concepts are introduced in such clear terms. However, I find the structure of Section 3 a bit convoluted. It takes a while to grasp how the pieces come together in the second and third paragraphs of Section 3.1. Further, the novelty claims are also a bit hidden. I recommend that the next revision of the paper includes a dedicated paragraph that explain in technical depth what exactly is different in the proposed solution from the state of the art. Lastly, presenting the whole method also as pseudocode in the main paper as Algorithm 1 would substantially improve the reader experience.

Experimental Evaluation
--

Both the baselines and the experiment setups are well-chosen and well-justified. However, the same is not the case for the evaluation metrics. The paper reports only the test log-likelihood in all experiments. While obviously being a useful score, it is not sufficient to draw conclusions about the prediction success by looking only at the log likelihood. The authors should definitely provide the plain RMSE scores of the same experiments in the rebuttal phase. Furthermore, the proposed method does not seem to improve the evaluated performance metric in the real-world data sets with respect to existing alternative solutions. It could well be that the proposed method has some properties more favorable than its competitors beyond prediction accuracy, but then there should be a corresponding experiment or evaluation metric that quantifies this favorableness.

Questions:
--

Fig 2 shows that in the data generating process, X_t depends on both Z_t and Z_{t-1}. Where does the dependency on Z_{t-1} come from? Which equation in the paper clarifies this dependency? Eq 6 is a non-Markoving mapping.

Overall
--

Interesting problem selection. The solution has some not groundbreaking but fairly novel aspects. Scientific writing is overall OK, but the novelty presentation (Sec 3) could be improved. Experimental evaluation is extremely weak. The solution falls behind multiple baselines. Evaluation wrt RMSE missing.
limitations_and_societal_impact: I do not find the answer of the authors to the checklist question 1 (b) convincing. Having closely read Section 3, I am not able to point out a clear discussion anywhere in the paper about the limitations of the "original solution" proposed by this work. The paper answers to some extent previously unsolved problem this paper has solved, but does not really talk about what its leaves unsolved or at the expense of what the solution has been achieved (e.g. increased computational cost, additional assumptions on the latent space, loss of interpretability etc.).
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper introduces a new form of generative model for continuous time-series consisting of a latent neural SDE transformed by a continuously indexed normalizing flow. This is motivated by the fact that continuously indexed normalizing flows with latent Brownian motions can only model a restricted class of SDE dynamics while neural SDE can only model Markov processes.
main_review: The paper addresses with the important problem of modeling discretely sampled continuous dynamics. The work is a development on recent work on continuous normalizing flows. Specifically, it can be seen as an improvement of the continuous-time flow processes [1]. 

The main difference between [1] and the current work is the use of a latent neural SDE instead of a the Wiener process as latent process. This greatly increases the representational flexibility of the approach since many Ito processes cannot be obtained by transforming a Weiner process with a bijector. All in all, the approach is natural improvement of [1] which fixes a clearly identifiable problem. The resulting method is however less elegant as it combines two separate parameterized components, each of which would work as stand-alone generative model. The novelty is limited since the approach is a natural combination of two popular methods without key new insights and model-specific derivations.  The paper also introduces a somehow novel variational inference method to infer the latent form the discretely sampled observations. 

Strengths:
- The paper addresses a clearly defined problem with important applications. It introduces a new method that solves some clearly stated limitation of state-of-the-art techniques. 

Weaknesses:
- The novelty of the approach is rather limited, being the direct combination of two existing methods. 
-The experiments are on the weak side for NeurIPS. I would have liked to see more experiments both on simulated timeseries and on real data. Given the setting, it would have also be informative to compare the result with Gaussian process and neural process baselines. The results are difficult to interpret since the paper does not provide error bars. The results on real-world data are rather underwhelming, especially for the CLPF-ANODE version, which is the one described in the methods section. There is a big difference between the continuous approaches and the much simpler VRNN in terms of likelihood. The proposed method does not seem to perform much better than the much simpler CTFP.

[1] Deng, Ruizhi, et al. "Modeling continuous stochastic processes with dynamic normalizing flows." 34th Conference on Neural Information Processing Systems


limitations_and_societal_impact: The limitations of their approach are only briefly discussed.
review_title: null

Meta-review: This paper explores a continuous latent time series model, the Continuous Latent Process Flow (CLPF). 

CLPFs can be seen as a generalisation of CTFPs where, instead of transforming a base Wiener process with a time-conditional flow, it first introduces a latent process SDE which conditions a CLPF-like model (that uses an OU base process instead of a Wiener process). In that sense, CLPFs combine elements from VRNNs, latent SDEs and CTFPs.

In terms of motivation/discussions, it is clear why CLPFs should be strictly better than CTFPs, but it is not clear why they should be better than latent SDEs (at least not "a priori") so adding more discussions on this could improve the draft.

The experiments show good performance on real-world data relative to CTFPs, latent SDEs and VRNNs. In spite of some hesitance amongst reviewers, I find the experimental results compelling. But I also agree that the paper could benefit from more detailed experiments comparing to VRNNs, for instance to clarify if the difference with respect to VRNN is due to implementation differences (e.g. passing absolute time or not, or small architectural differences).
Decision: Accept (Poster)