Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper builds on ODE-RNN model that allows to represent a time series as a continuous trajectory. The authors address the limitation of the ODE-RNN model that the trajectory is continuous everywhere except the observation points. They introduce a compensation term based on cubic splines that transforms the output trajectory into a continuous one. This approach is also applied to correct a hidden state trajectory. The authors demonstrate better interpolation properties on sparse time series data.

Pros:
+ The paper targets the main drawback of ODE-RNN that both the output and the hidden state have a "jump" at the observation points. The idea of correcting the trajectory or hidden state using cubic splines to make it continuous is novel and interesting.

+ The authors provided the theoretical justification for their method and an interpolation error bound. The authors provided the thorough analysis of the model, including the effect of the numerical differentiation.

+ The authors highlighted the two cases of input data (with the continuity on the input level and on the semantic level) and demonstrated the applicability of CSSC and hidden CSSC in each case. 

+ In hidden CSSC, the trajectory is being smoothed out without relying on ground-truth data.

Cons:
- I find the decision to set the outputs exactly to x_k  in standard CSSC to be questionable. Instead of correcting the model predictions, like in hidden CSSC, the model explicitly sets the the predictions to the correct value. First, as authors mention, it restricts the model to the time series with no noise, which has a limited use in practice. Second, with incorrect alpha, the computational capacity is of ODE-RNN is not utilized, and the model reverts to a cubic spline to fit the data. It is visible on figure 5 (alpha <= 10) -- the corrected output \hat(o)(t) fits the trajectory well despite the poor reconstruction by ODE-RNN.
Is it possible to do the interpolation in the input space by setting epsilon_k+ = 0; epsilon_k- = o(t_k+) - o(t_k-), similarly to the hidden CSSC? Then the correction c(t) will only play a role of closing the gap between o(t_k+) and o(t_k-). The output of CSSC will remain continuous, but the outputs are no longer restricted to match x_k, improving the expressive power of the model.

- The paper does not provide the error bars for the quantitative results. It is hard to make conclusions about advantages of CSSC and hidden CSSC over other models.

- The proof of interpolation lower bound requires x(t) to have continuous derivatives up to the forth order. It is quite restrictive. I believe that trajectories from Mujoco and moving MNIST already violate this assumption. Can authors hypothesize in which applications we might be able to assume that the inputs belong to C^4?

- Unlike ODE-RNN, the model cannot be used for extrapolation or in the online setting, because the computation of c_k(t) involves the next time point k+1.

Other comments:

-Was the set of subsampled inputs fixed? Was the same set of sampled points used for all the models? In the experiments, only a small fraction of the time points is used as input, and the model performance may change drastically based on the exact set of sampled points. 

-I wanted to mentioned that there is a related work by Kidger et al., NeurIPS 2020 "Neural Controlled Differential Equations for Irregular Time Series". They use cubic splines on the input data, which allows Neural ODE to query the data at any time point and produce a smooth hidden state trajectory.  The current work achieves a similar goal without modifying the inputs.

-In Table 1 in the first column (Toy data set with 10% observations) Latent ODE actually has the lowest MSE of 0.013768, while CSSC has 0.024656.

-Section 4.5 "While standard CSSC can always increase the performance upon ODE-RNN". The increase in performance is not guaranteed. I suggest changing the wording to "Standard CSSC has higher performance than ODE-RNN in all our experiments"


Overall:

I vote for weak acceptance of the paper. The idea of adding a compensation term resolves the important issue of the ODE-RNN model and is valuable to the ICLR community. I appreciate the idea of hidden CSSC, where only the predicted hidden states were used to produce a smooth trajectory in the latent space. In standard CSSC, my concern is that the predictions at observation times \hat(t_k) are explicitly set to x_k, making the model inapplicable to noisy data in real-world applications.

I would like to see the error bars for table 1, where the models were trained on the exact same set of subsampled inputs, but with different random seeds. It will help to understand the gap between the cubic splines and CSSC.

============================================

Added after the author response:

The authors adequately addressed my questions and concerns. I appreciate that the authors provided the error bars for their experiments and tried out the idea of modifying epsilon_k, but I wish the authors left more time for the discussion. I think the paper is a valuable contribution to the domain of irregular time series. I am increasing my score to 7. 
review_title: Resolving the discontinuity of ODE-RNN outputs using splines

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes to use cubic spline interpolation to smooth out discontinuities outputs and state inferred by ODE-RNNs.

While the approach is interesting, it is ill adapted to noisy data (which is a very strong shortcoming IMO) while the experiments are limited to data sets corresponding to applications with limited impact due to their artificial nature.

The paper has multiple issues which I believe require serious rewriting.

1) The paper has grammatical mistakes including in the abstract and many run-on sentences. The readers could use higher quality writing.

2) The premise that the jumps inside the ODE-RNN are shortcomings or inconsistencies is erroneous. It is expected that their would be jumps with each new observation as the filtration with respect to which the latent state is conditioned changes then with a discontinuity. It would in fact be unexpected to have a continuous posterior. This is for instance what typically happens to stock market stock series any time an earnings report is issued.

3) The experimental section is terse and could really use more diverse datasets. Also comparisons with respect to Gaussian and Neural processes would be welcome. A comparison with a simpler smoothing method such as a convolutions, Fourier and wavelet denoising is also warranted.
review_title: Interesting paper but technique has strong limitations while writing could be improved

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This work addresses the discontinuity issues caused by jumps in hidden state/output at the arrival of new observations in a ODE-RNN. This problem is tackled by adding a cubic spline smoothing component on top of ODE-RNN to produce smooth and continuous hidden state/outputs. They derive a closed form solution for the cubic spline component based on the output of ODE-RNN and obtain an error bound for 4th order derivable inputs. Although the cubic spline component has no trainable component, this work shows that the gradient can flow through it to ODE-RNN and perform end-to-end training. 

Positives:
1. The paper focus on the task of learning from irregularly sampled data which is important in many domains. 
2. The paper is well written and easy to follow.
3. Experiments show that the model indeed learns to perform continuous interpolation.

Concerns:
1. The key concern about the paper is the lack of rigorous experimentation to study the usefulness of the proposed method. The authors mention that this approach would be most useful in continuous time series interpolation for irregularly sampled time series , but there have been no experiments on real world irregularly sampled time series (e.g. PhysioNet). 
2. Another concern I have is with the performance of Latent ODE baseline. The results in Rubanova et al. show that Latent-ODE model almost always outperforms ODE-RNN but that is not the conclusion I get from the results in this paper. This makes me question the validity of implementation/experiments.
3. ODE-RNN can also be used for extrapolation in addition to interpolation. Is it possible to perform extrapolation using the proposed model? 

Additional Comments:
1. Could the authors comment on the performance of a Latent ODE model with CSSC as encoder and ODE as decoder? 
2. It would be interesting to see if the improved interpolation leads to improved performance in downstream tasks such as sequence classification.
2. Missing comparison with kernel based interpolation methods e.g. Shukla and Marlin (2019).
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper presented how cubic spline smoothing function was used to compensate the insufficiency in ODE-RNN (ordinary differential equation recurrent neural network) for irregularly sampled sequences.

Pros: 
This paper presented a nice solution to enhance the performance of ODE-RNN with mathematical evidence. Theoretical justification was provided. The solution was evaluated in different tasks.

Cons: 
This solution required the calculation of inverse matrix in a compensation term which would cause a significant increase in training time when the dimension is enlarged.
review_title: A nice solution to insufficiency in neural differential equation by introducing the cubic spline smoothing compensation

Meta-review: This paper introduces a form of cubic smoothing for use with ODE-RNNs, to remove the jump when new observations occur.  I think this paper's motivation is based on a misunderstanding of what the hidden state of an RNN represents.  Specifically, an RNN hidden state is a belief state, not the estimated state of the system.

I think R2 is right that it's correct for a filter to jump when seeing new data.   It's not a matter of whether the phenomenon being modeled is slow-changing or not.  The filtering output is a belief state, which can change instantaneously even if the true state does not.

The important distinction to make is filtering (conditioning only on previous-in-time data) vs smoothing (conditioning on all data).  The smoothing posterior should generally be smooth if the true state changes slowly.

As R4 notes, all of the tasks are based on interpolation, which is not what the ODE-RNN is trying to do, and the proposed method would make the same predictions as a standard ODE-RNN.  Finally, as R4 notes, "The authors do not provide any experimentation on real-world irregularly sampled time series".
Decision: Reject