Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper introduces Variational Stochastic Differential Networks to filter and smooth sporadically observed time series.

The authors adopt a Bayesian perceptive on the smoothing problem for time series living a latent space and irregularly observed.
In particular, the random evolution of the process in latent space is clearly accounted for in the paper by embedding an SDE into an RNN.

The authors derive a variational loss for their model and describe in fact multiple losses with different improvements such as importance sampling. In the end the authors do report issues caused by excess variance in training and settle for a convex combination of the two losses they propose.

The paper becomes quite interesting in its experimental part as the authors show the superiority of their method as compared to previous approaches on data sets concerned with Mocap, synthetic OU process data and meteorological data.
review_title: Good paper yielding an interesting sequential SDE ELBO

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper introduces a latent variable model for high dimensional stochastic time-series. The model is akin to a VAE with RNNs that incorporate time-series data. The authors introduce two variants of the model, one which only contains a feedforward RNN (filtering) and another that contains feedforward and feedback RNNs (smoothing).  The authors use two inference procedures for the model, one the standard VAE, and the other importance weighted IWAE.  The work is reasonably clearly presented and the experiments are multiple data sets are a nice addition.


I have two primary concerns -- 1) I do not see how the choice of the two inference procedures (which makes up a fair portion of the submission) is well motivated, and  2) I find the arguments for them achieving state-of-the-art performance unconvincing. 

for 1)
The use of these two inference methods feels ad hoc. Why not just choose one? Using importance waiting and the standard ELBO does not seem to add to the paper. I see the primary contribution of the paper to be the model, not the inference procedure. If there is an important distinction between the inference methods, and the authors feel that is important to presenting their work, I would devote more space to explaining why there are two approaches --  the details of the objectives themselves could be moved to an appendix if needed.

Additionally, regarding the inference procedure, it looks like a beta-VAE objective is used (equation 6), but the authors do not say why the hyperparameter beta is needed, and do not cite the Beta-VAE paper (Higgins et al. 2017)

For 2)  I would like to see more comparing the performance of this model to a stochastic high dimensional. The authors discuss neural ODEs, but there is no comparison against them for performance in the experiment section. If the comparison cannot be made for some specific reason, the authors should explain why.

Another model which comes to mind that might warrant some discussion/comparison is lfads (Sussillo et al. 2016). 

One other point: I feel similarly to the choice of two models (VSDN-F and VSDN-S) as I do about the IWAE and VAE. I think it would be better to hone in on a stronger take home point -- do one of the methods (say, VSDN-S) achieve better performance than existing methods. Showing this on more tasks or compared to more models would, in my mind, make this a stronger submission.

Some other concerns:

It is hard to get anything out of figure two. The human skeletons are very small and there are many of them. If the authors wish to show how the distinct positions for each of the methods differ from ground truth, they should remove some of the panels (again, not sure here why the different inference procedure warrants a different panel. I don't think the authors want to communicate how the different inference procedures make an important difference, do they?) An additional task or more clear plot showing accuracy of recovered latents, or accuracy of generated time-series data would be useful.


Typo in the final sentence of the first paragraph of the experiments section. You say "two" but compare to three things.
review_title: Review of Learning Continuous-Time Dynamics by Stochastic Differential Networks 

final_score: rating: 7: Good paper, accept
review_content: This paper aims to model the complicated continuous time-series by using SDE for modeling the latent state trajectories. The authors claim that using SDE instead of ODE for the latent states has higher flexibility to capture more complex dynamics. Then they propose a continuous-time versions of variational evidence lower bounds (ELBO) which can be trained using ODE-RNNs as the inference networks. The proposed VSDN model has the capability to capture the latent state stochasticity not via the initial states but rather via SDEs, as opposed to other methods like latent ODE and latent SDE. 

Overall, the paper is interesting as it employs many recent advances in the field to introduce a richer model for the continuous complex time-series with the capability of taking into account the stochasticity of the latent state dynamics. Nevertheless, I have some questions as follows:

1- The first motivation of the paper to define the VSDN is that the methods based on the neural ODEs cannot model complicated time-series. I wonder why is this claim true? When the underlying latent state is an ODE, the stochasticity of the time-series could be modeled via rich conditional observation distributions. Why do we need to have a double stochasticity, one in latent state and one in the observation data?

2- How many dimensions can VSDN handle? I am wondering how it will work in a high dimensional regime (say dim is several thousands). How many dimension are used for states and observation data?

3- As the VSDN is using SDEs for the latent states, I wonder if it can quantify the uncertainty of the inferred latent state trajectories? 

4- Why in eq. (13), $h_{t,pre}$ and $h_t$ are summed? Could they be concatenated instead of summation? 

5- Minor comment: I guess $W_t$ in eq. (2) refers to the Wiener process, right? But there is no definition on the text. So, the authors should define it in the paper so that the readers know what it is. 


########## EDIT ##########
The authors have addressed all my questions.
review_title: This paper proposes a new continuous-time stochastic recurrent network called Variational Stochastic Differential Network (VSDN) that incorporates Stochastic Differential Equations (SDEs) into recurrent neural model to effectively model the continuous-time stochastic dynamics based only on sparse or irregular observations.

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper the author/s study/ies the fundamental problem of learning continuous-time stochastic dynamics, in the case where the available time series data suffer irregularity and sparseness. The paper assumes that high dimensional data are generated from a system where latent states are observed. The paper tackles the problem 
In such a settings, it is generally impossible to derive a continuous-time stochastic process which precisely describes the many behaviors of the system under study.
The paper develops a method base on Variational Bayes applied to learn flexible continuous-time stochastic recurrent neural network, they call Variational Stochastic Differential Networks. The particualr feature of such model is to capture the stochastic dependency among latent states and observations.
The paper provides theoretichal tool, under the form of lower bounds for the efficient training of the neural model.
THe author/s state that the results from a rich set of numerical experiments witnesses that the proposed approach 
outperforms state-of-the-art continuous-time deep learning models to solve the prediction and interpolation tasks, in the particular case when irrelugar and sporading time series data is available.
----------------------------------------------------------------------------------------------------------------------

Reason for Score:
Overall, I vote for rejecting. I like the idea to mix parametric and non parametric components to learn time series dynamics. However, I have many concerns that are made explicit in the cons section.
Hopefully the authors can address my concerns in the rebuttal period. 

----------------------------------------------------------------------------------------------------------------------
Pros.
1) the paper tackles a relevant problem.
2) I appreciate the idea to combine parametric and non parametric models
3) the proposed model is clear
3) numerical experiments, on the selected data sets, witness in favour of the proposed method
----------------------------------------------------------------------------------------------------------------------

Cons.
1) why other models for time series modeling and forecasting have not been taken into accout? what about dynamic Bayesian networks, hidden Markov models, contiuous time Bayesian networks, and many other ...

2) the paper states the interest is related to high dimensional time series, but the data sets used for numerical experiments, in my humble opinion, are not as such, they are of very small or moderate dimensionality, in terms of number of variables.

3) I found no discussion about the choice of the dimension of the latent space, maybe my fauls and if this is the case I apologize, but still I think that more attention and more motivations must be given in the numerical experiments section.

4) I'm under the impression that the proposed solution is a sensible combination of existing results and thus the work is somewhat of incremental nature, which I do not know whether is relevant for ICLR. 

5) I'm confused on the formulation of the inference problems, the paper considers both filtering and smoothing. However, later in the paper, the numerical experiments section, I read about "prediction" and "interpolation" which in my uderstanding is the same as filtering and smoothing. Thus, I would like the authors to better frame their problem.

6) at page 2, I can not understand the difference between path and history, if they are used as synonims, please do not do it, it increases confusion to the interested reader, try to be more formal and clear as possible to let interested reader to appreciate your contributions.

7) at page 3 I read "However, RG only use the historical data as input, as we observe no improvement by including the
current state.", I would ask to motivate, personally I'm not that surprised by this but I would like the paper to help understand the why of this occuring.

8) In equation (5) you are using the entire trajectory and thus you are performing smoothing, I think the problem/s tackled must be clearly described.

9) at page 5 I read "In real-world applications, the latent state may not have strong dependency on future observations". This is somewhat surprising me, which has not strong dependency on which? the latent does not influence the future? the future does not tell much about latent? This sentence in obscure to my elementary mind, and thus I kindly ask to better clarify on it.

10) Table 1 and 2, please call the macro column filtering and smoothing in place of prediction and interpolation, maybe I completely miss the meaning of the difference you made between prediction and filtering and the difference between smoothing and interpolation. If this is the case, please better clarify on these aspects.

11) NLL and MSE of VSDN-S (VAE) are always better that those achieved by VSDN-F (VAE), this is not that strange, one is using more info than the other, as well as I understood. The same does not apply to VSDN-S (IWAE) and VSDN-F (IWAE), and this is surprising in my humble opinion, maybe not found the optimum?

12) In table 2, NLL is positive for LatentSDE, can you comment on this?

13) The same as 12) in Table 3, I kindly ask to clarify under which circumstances this happens.
----------------------------------------------------------------------------------------------------------------------
Questions during rebuttal period: 

Please address and clarify the cons above 
----------------------------------------------------------------------------------------------------------------------
Some typos: 

page 3: I read "However, RG only use the historical", it should read "However, RG only uses the historical"
        I read "HQ also use the", it should read "HQ also uses the"
----------------------------------------------------------------------------------------------------------------------


review_title: Need to better understand different aspects of the paper

Meta-review: This paper sits right at the borderline: the reviewers agree that it is interesting and addresses a relevant problem. On the negative side, the presentation could be improved (including some incorrect claims), and the experiments could be strengthened (both in terms of baselines and datasets used). Ultimately, the paper will probably require another round of reviews before it is ready for publication.
Decision: Reject