Cubic Spline Smoothing Compensation for Irregularly Sampled Sequences
Authors: Jing Shi
Jing Bi
Yingru Liu
Chenliang Xu
Venue:  (2021)
Keywords: Neural Ordinary Differential Equations
Cubic Spline Interpolation
Irregular Time Series

Abstract: The marriage of recurrent neural networks and neural ordinary differential networks (ODE-RNN) is effective in modeling irregularly sampled sequences.
While ODE produces the smooth hidden states between observation intervals, the RNN will trigger a hidden state jump when a new observation arrives and thus cause the interpolation discontinuity problem.
To address this issue, we propose the cubic spline smoothing compensation, which is a stand-alone module upon either the output or the hidden state of ODE-RNN and can be trained end-to-end.
We derive its analytical solution and provide its theoretical interpolation error bound.
Extensive experiments indicate its merits over both ODE-RNN and cubic spline interpolation.

Under review as a conference paper at ICLR 2021
CUBIC SPLINE SMOOTHING COMPENSATION FOR IRREGULARLY SAMPLED SEQUENCES
Anonymous authors Paper under double-blind review
ABSTRACT
The marriage of recurrent neural networks and neural ordinary differential networks (ODE-RNN) is effective in modeling irregularly sampled sequences. While ODE produces the smooth hidden states between observation intervals, the RNN will trigger hidden state jump when a new observation arrives and thus cause the interpolation discontinuity problem. To address this issue, we propose the cubic spline smoothing compensation, which stand-alone module upon either the output or the hidden state of ODE-RNN and can be trained end-to-end. We derive its analytical solution and provide its theoretical interpolation error bound. Extensive experiments indicate its merits over both ODE-RNN and cubic spline interpolation.
INTRODUCTION
Recurrent neural networks (RNNs) commonly used for modeling regularly sampled se- quences (Cho et al. 2014). However, the standard RNN can only process discrete series without considering the unequal temporal intervals between sample points, making it fail to model irregu- larly sampled time series commonly seen in domains, e.g., healthcare (Rajkomar et al., 2018) and finance (Fagereng& Halvorsen| 2017). While some works adapt RNNs to handle such irregular scenarios, they often assume an exponential decay (either at the output or the hidden state) during the time interval between observations (Che et al.) 2018 Cao et al.) 2018 which may not always hold.
To remove the exponential decay assumption and better model the underlying dynamics. Chen et al. (2018) proposed to use the neural ordinary differential equation (ODE) model the continuous dynamics of hidden states during the observation intervals. Leveraging a learnable ODE parametrized by a neural network, their method renders higher modeling capability and flexibility.
However, ODE determines the trajectory by its initial state, and it fails to adjust the trajectory according to subsequent observations. A popular way to leverage the subsequent observations is ODE-RNN (Rubanova et al. 2019; De Brouwer et al. 2019). which updates the hidden state upon observations using an RNN, and evolves the hidden state using an ODE between observation intervals. While ODE produces smooth hidden states between observation intervals, the RNN will trigger hidden state jump at the observation point. This inconsistency (discontinuity) is hard to reconcile, thus jeopardizing continuous time series modeling, especially for interpolation tasks (Fig. top-lefft)
We propose a Cubic Spline Smoothing Compensation (CSSC) module to tackle the challenging discontinuity problem, and it is especially suitable for continuous time series interpolation. Our CSSC employs the cubic spline as a means of compensation for the ODE-RNN to eliminate the jump, as illustrated in Fig. tigg-. While the latent ODE (Rubanova al. 2019) with an encoder-decoder structure can also produce continuous interpolation, CSSC can further ensure the interpolated curve pass strictly through the observation points. Importantly, we can derive the -oored-fomm solution for CSSC and obtain its interpolation error bound. The error bound suggests two key factors for a good interpolation: the time interval between observations and the performance of ODE-RNN. Furthermore, we propose the hidden CSSC that aims to compensate for the hidden state of ODE-RNN (Fig. M bottom), which not only assuage the discontinuity problem but is more efficient when the observations are nigh-dimensional and only have continuity on the semantic level. We conduct extensive experiments and ablation studies to demonstrate the effectiveness of CSSC and hidden CSSC, and both of them outperform other comparison methods.

---

Under review as a conference paper at ICLR 2021
ODE-RNN
Our methods
Smoothed output 0
Ground truth Output
Compute
CSSC
8 0=0+c
Time
11
fe
1g
F2
Output
Ground truth Output
Compute hidden CSSC c
hm hmh+c
Hidden state
Smoothed hidden state
Time
10
Figure 1: The illustration of ODE-RNN and our methods. The top left is ODE-RNN showing the interpolation curve jump the observation points. The top right is the smoothed output by our CSSC, where the jump is eliminated, and the output strictly passes the observation. The bottom left shows the ODE-RNN 's discontinuous output caused by the hidden state discontinuity. The bottom right shows that our hidden CSSC is applied to the hidden state of ODE-RNN, resulting in the smooth hidden state and so as the output.
RELATED WORK
Spline interpolation is practical way to construct smooth curves between number of points (De Boor et al. 1978), even for unequally spaced points. Cubic spline interpolation leverages the piecewise third order polynomials avoid the Runge's phenomenon (Runge 1901) and is applied as classical way to impute missing data (Che et al. 2018).
Recent literature focuses on adapting RNNs model the irregularly sampled time series, given their strong modeling ability. Since standard RNNs can only process discrete series without considering the unequal temporal intervals between sample points, different improvements were proposed. One solution is to augment the input with the observation mask or concatenate it with the time lag At and expect the network to use interval information At in an unconstrained manner (Lipton et al.) 2016, Mozer et al. 2017). While such flexible structure achieve good performance under some circumstances (Mozer et al. 2017), a more popular way is to use prior knowledge for missing data imputation. GRU-D (Che et 2018) imputes missing values with the weighted sum of exponential decay of the previous observation and the empirical mean. (hhukla & Marlin (2019) employs the radial basis function kernel to construct an interpolation network. Cao al. (2018) let hidden state exponentially decay for non-observed time points and use bi-directional RNN for temporal modeling.
Another track is the probabilistic generative model. Due the ability model the missing data's uncertainty, Gaussian processes (GPs) are adopted for missing data imputing (Futoma et al.) 2017] Tan et al., 2020 Moor al. 2019). However, this approach introduced several hyperparameters, such as the covariance function, making it hard to fine-tune in practice. Neural processes (Gamelo et al.) 2018) eliminate such constraints by introducing a global latent variable that represents the whole process. Generative adversarial networks are also adopted for imputing (Luo et al. 2018).
Recently, neural ODEs (Chen al.) 2018) utilize a continuous state transfer function parameterized by neural network to learn the temporal dynamics. Rubanova et al. (2019) combine the RNN and ODE to reconcile both the new observation and latent state evolution between observations. De Brouwer al. (2019) update the ODE with the GRU structure with Bayesian inference observations.
While the ODE produces the smooth hidden states between observation intervals, the RNN will trigger jump of the hidden state at the observation point, leading to a discontinuous hidden state along the trajectory. This inconsistency (discontinuity) is hard to reconcile, thus jeopardizing the modeling of continuous time series, especially for interpolation tasks. The neural CDE (Kidger
2

---

Under review as a conference paper at ICLR 2021
al. 2020) directly apply cubic splines interpolation at the input sequence make the sparse imput continuous and thus produce continuous output. On the contrary, our method tackles this jumping problem by introducing the cubic spline compensation for the vanilla ODE-RNN, at either the output or hidden space.
3 METHODS
In this section, we first formalize the irregularly sampled time series interpolation problem (Sec. 3.1 then introduce the background of ODE-RNN (Sec.(3.2) Based upon ODE-RNN, we present CSSC and its closed-form solution (Sec. 3.3 and illustrate the inference and training procedure (Sec. 3.4) Finally, we provide the interpolation error bound of CSSC (Sec. 3.5) and describe an useful extension of CSSC (Sec. (.6..3.6).
3.1 PROBLEM DEFINITION
We focus the interpolation task. Given an unknown underlying function x(I) :R - R4 E (a, b). and a set of n + observations [xx|Xx> = xl++)]==o IR sampled from x(1) the irregularly spaced time points IT a fo < l < ... < bn = b, the goal is to learn a function F(t) R RRd to approximate such that Ft&)) = Xg.
3.2 BACKGROUND OF ODE-RNN
ODE-RNN (Rubanova et al. 2019) achieves the interpolation by applying ODE and RNN inter- changeably through a time series, illustrated in top-left of Fig. The function on time interval t [kk (4+1) is described by neural ODE with the initial hidden state h(4)):
(tt) = f(((t)); (() g((()),
(2)
where the h E R. is the hidden embedding of the data, h = dh the temporal derivative of the at hidden state, E Rd is the interpolation output of F(t). Here, f I + R" and R" + Rd are the transfer function and the output function parameterized by two neural networks, respectively. At the observation time / = 4, the hidden state will be updated by RNN as:
h(++) RNNCcI(((( x:);; (t2)) = gh(())))
(3)
where the input E IRd, 1 and 1 are the left- and right-hand limits of 4g. The above formulation has two downsides. The first the discontinuity problem: while the function described by ODE is right continuous o(t,) o(tt)) ), the RNN cell in Eq. (3) renders the hidden state discontinuity h(()) A h((*)) and therefore output discontinuity o(t,) o o(()). The second is that the model cannot guarantee o(tx) = X2 without explicit constraints.
3.3 CUBIC SPLINE SMOOTHING COMPENSATION
To remedy the two downsides, we propose the module Cubic Spline Smoothing Compensation (CSSC), manifested in the top-right of Fig. It computes a compensated output ö(l) as:
6(t) cct) + o(t),
(5)
where o(t) is the ODE-RNN output, and the c(t) is compensation composed of piecewise continu- ous functions. Our key insight is that adding another continuous function to the already piecewise continuous (t)) will ensure the global continuity. For simplicity, we set c(t) as a piecewise polyno- mials function and then narrow
piecewise cubic function since is the most commonly used polynomials for interpolation (Burden & Faires 1997). As the cubic spline is computed for each dimension of e individually, w.l.o.g., we will discuss one dimension of the o, C, Ô, x and thus denote them as o, c, ô, x. respectively. c(t) is composed with pieces as c(t) = Σ Cj (t) with each piece c defined at domain [4,, (1). To guarantee the smoothness, we propose four constraints o(l):
3

---

Under review as a conference paper at ICLR 2021
1. o(t)) o(ti)) 2g, lt 1,... 1, 0(to) 2o. ō(tn) 2n (output continuity);
2. )(tz) =((t)), ) k -,..., (first order output continuity);
3. o(tz) =(tt)) ) k 1...., (second order output continuity);
4. ö(to) ö(t..) 0 (natural boundary condition).
The constraint 1 ensures the interpolation curves continuously pass through the observations. Con- straint 2 and 3 enforce the first and second-order continuity at the observation points, which usually holds when the underline curve is smooth. And constraint 4 specifies the natural boundary condition owing to the lack of information of the endpoints (Burden & Faires 1997).
Given ott) and such four constraints, Theorem 1. Given the first order and second order jump difference of ODE-RNN as
c(t) has unique analytical solution expressed in Theorem 1
Ax = -6( );
(6)
ř& = - ö(tz ).
(7)
where the analytical expression of à and ö can be obtained as
0g oh
ö ST 8gg g
0g T 0f Th 8h J
s:
(8)
Hh
and the error defined as
c Jy - o((x);
(9)
E I o(tx)) )
(10)
then Cg can be uniquely determined
MR+1 + Fk411 Mr (1 1x))
Mr. (t )x
(1)
6T
2
k+1
x
TM + Fk++1 + 2M
[t - tb) + E
(1I)
Th
where Mj is obtained as
M = A-
(12)
2
A1
Mi
di
P/2
2
X2
M2
d2
A =
.M =
dd =
:
(13)
Pn-2
An-
Mn
dn-2
Mn-1
2
M
d 1/
Th = 1441 - t4. 14k
Tk-1
2 =
Th
1 d2 = 6
elle elti ((41) -elt
6/2
2F
-Pe41
To 1
1
1
elti t
Mo = Mn = 0.
Tk
The proof for Theorem. is in Appx. The c(t) is obtained by computing each c(t) individually
according to Theorem.
Computational Complexity. The major cost the inverse of A, a tridiagonal matrix, whose inverse can be efficiently computed in ((n) complexity with the tridiagonal matrix algorithm (implementation detailed in Appx. [.I). Another concern is that and ö needs to compute Jacobian and Hessian in Eq. g We can circumvent this computing cost by computing the numerical derivative or an empirical substitution, detailed in Appx. C.2
Model Reduction. Our CSSC reduce to cubic spline interpolation if setting in Eq. 5) zero In light of this, we further analyze our model with techniques used for cubic spline interpolation and experimentally show our advantages against it in Sec. 4.2
4

---

Under review as a conference paper at ICLR 2021
3.4 INFERENCE AND TRAINING
For inference, firstly compute the predicted value o from ODE-RNN (Eq. (14) then calculate the compensation with CSSC(Eq. (IT); thus yielding smoothed output (Eq. 5 For training. the CSSC is a standalone nonparametric module (since we have its analytical solution) on top of the ODE-RNN that allows the end-to-end training for ODE-RNN parameters. We employ Mean Squire Error (MSE) loss to supervise ó. In addition, we expect the compensation c to be small to push ODE-RNN to the leading role for interpolation and take full advantage of its model capacity. Therefore 2-norm penalty for с is added to construct the final loss:
1 E ((x(t))) o(+))l + N
L
(14)
The ablation study (Sec. of ODE-RNN and CSSC.
4.5) shows that the balance weight a can effectively arrange the contribution
Gradient flow. Although is non-parametric module. but the gradient can flow from it into the ODE-RNN because c(t) depends on the left and right limit of o(tx), o(tx), ö(tx). We further analyze that ó(tx) plays a more important role than (f+)) in the contribution to c(t), elaborated in Appx. C.3
3.5 INTERPOLATION ERROR BOUND
With CSSC, we can even derive an interpolation error bound, which is hard to obtain for ODE-RNN. Without loss of generality, we analyze one dimension of ó, which is scalable to all dimensions.
Theorem 2. Gäven the CSSC at the output space as Eq. (5) if Ca, I C C then the error and the first order error are bounded
[| - 110e S CCll((x - where || Ilso is uniform norm, (-)r r-th derivative, Co 384 C1 24' T is the maximum interval over II.
(r= 0, 0,1),
(15)
The proof of Theorem in Appx. B The error bound guarantee the error can converge zero if - or [I - o) (IT] + O. This suggests that better interpolation can come from denser observation or a better ODE-RNN output. Interestingly, Eq. (15) can reduce to the error bound for zero. Compared with ODE-RNN, which lacks the convergence guarantee for T, our model more effectively mitigates the error for the densely sampled curve at complexity O(7*)= compared with the cubic spline interpolation, our error bound has an adjustable o that can leads smaller [[ - (| than ||x((1)
cubic spline interpolation (Hall & Meyer 1976) if o
An implicit assumption for this error bound is that the x should be 4-th order derivable; hence this model is not suitable for sharply changing signals.
3.6 EXTEND INTERPOLATE IIDDEN STATE
Although CSSC has theoretical advantages from the eтror bound, it is still confronted with two challenges. Consider the example of video frames: each frame is ligh--imensional data, and each pixel is not continuous through the time, but the spatial movement of the content (semantic manifold) continuous. So the first challenge is that CSSC has linear complexity w.r.t. data dimension, which is still computation demanding when the data dimension becomes very high. The second challenge is that CSSC assumes the underlying function x is continuous, and it cannot handle the discontinuous data that is continuous in its semantic manifold (e.g. video).
To further tackle these two challenges, we propose a variant of CSSC that is applied to the hidden states, named as hidden CSSC, illustrated in bottom of Fig. D As only compute the compensation to hidden state, which can keep a fixed dimension regardless the how large the data dimension is, the computational complexity of hidden CSSC is weakly related data dimension. Also, the hidden state typically encodes a meaningful low-dimensional manifold of the data. Hence smoothing the hidden state is equivalent to smoothing the semantic manifold to certain degree.
Hence, the Eq. 5 is modified in(t) c(l) + h((), and the output (Eq. (..4 to o() - (h(())
However, since there is no groundtruth for the hidden state like the constraint 1, assume the h(tt)
5

---

Under review as a conference paper at ICLR 2021
22
GT
096
13.
CLAK
1--
10-
1.2
110
24
0.-
20.
Comparison methods
CSSC
(c) Hidden CSSC
Figure 2: The visual result for Sinuous Wave dataset with 10 observations. (a) shows the comparison of CSSC against other methods; (b) demonstrates the effect of с to smooth the of (c) shows the hidden CSSC output is smooth because its unsmooth hidden state h is smoothed into h.
the knots passed through by the compensated curve, rendering the constraints 1 as h((,)) h(tt)) h(tt)) The rationality for the knots is that h(t,)) updated by x(tx) and thus contain more information than h(t,)) Given the above redefined variable, the closed-form solution of is provided in Theorem. 3
Theorem 3. Given the second order jump at hidden state as
F&K h (i(tzz) h((t);;
Fs = b(tt)) h((t))
(16)
where h S, Of Th J, and the error defined as
c h(t))) h(tt)) O:
c h((t)) h(t)),
(17)
the Ck is uniquely determined as Eq. 777.
Theorem 3 suggests another prominent advantage: hidden CSSC be more efficiently implemented because its computation does not involve Hessian matrix.
4 EXPERIMENTS
4.1 BASELINE
We compare our method with baselines: (1) Cubic Spline Interpolation (Cubic Spline) (2) A classic
RNN where At is concatenated to the input (RNN-At) (3) GRU-D (Che et al., 2018) (4) ODE- RNN (Rubanova al. 2019) which is what our compensation adds upon (5) Latent- (Rubanova) et al. 2019) which is VAE structure employing ODE-RNN as the encoder and ODE as decoder (6) GRU-ODE-Bayes (De Brouwer et al. (2019) which extends the ODE as a continuous GRU, Our proposed methods are denoted as CSSC for data space compensation and hidden CSSC for hidden
space compensation. Implementation Details is in Appx. D
4.2
ToY SINUOUS WAVE DATASET
The toy dataset is composed of 1,000 periodic trajectories with variant frequency and amplitude. Following the setting of Rubanova et al. (201)), each trajectory contains 100 irregularly time points with the initial point sampled from standard Gaussian distribution. Fixed percentages of observations are randomly selected with the first and last time points included. The goal is to interpolate the full set of 100 points.
The interpolation error on testing data is shown in Table where our method outperforms all baselines different observation percentages. Figure 2 (a) illustrates the benefit of the CSSC over cubic spline interpolation and ODE-RNN. Cubic spline interpolation cannot interpolate the curve when observations are sparse, without learning from the dataset. ODE-RNN performs poorly when jump occurs at the observation. However, CSSC can help eliminate such jump and guarantee smoothness at the observation time, thus yielding good interpolation. In Figure (b). we visualized the ODE-RNN output and compensation c, and the CSSC output ó in detail to demonstrate how the the bad o becomes a good by adding c. Finally, Fig, (c) demonstrates the first dimension of hidden states before and after smoothing, where the smoothed hidden state leads smoothed output.

---

Under review as a conference paper at ICLR 2021
Table 1: Interpolation MSE toy, MuJoCo, Moving MNIST test with different percentages of
observations.
Toy
MuJoCo
Moving MNist
Observation
10%
30%
50%
10%
30%
50%
20%
30%
Cubic Splinc
0.801249
0.003142
0.000428
0.016417
0.000813
0.000125
0.072738
0.252647
RNN-AA
0,449091
0.245546
0.102043
0.028457
0.019682
0.008975
0,040392
0.037924
GRU-D
0,4739954
0.247522
0.121482
0.056064
0.018285
0.008968
0,037503
0.035681
Latent-ODE
0.013768
0.002282
0.002031
0.010246
0.009601
0.009032
0.035167
0.031287
GRU-ODE-Bayes
0.117258
0.010716
0.000924
0.028457
0.006782
0.002352
0,034093
0.030975
ODE-RNN
0,025336
0.001429
0.000441
0.011321
0.001572
0.000388
0.022141
0.016998
hidden CSSC
0.027073
0.001 503
0.000244
0.004554
0.000378
0.000106
0.019475
0.015662
CSSC
0,024656
0.0004557
0.000128
0,006097
0.040375
0.000087
GT
Spline
ODE-RNN
Figure 3: The visual result for MuJoCo. We visualize part of the interpreted trajectory with 5 frames interval. 10% frames are observed out of 100 frames. The observation is highlighted with white box.
4.3 MuJoCo PHYSICS SIMULATION
We test our proposed method with the "hopper" model provided by DeepMind Control Suite Tassa al. 2018) based on MuJoCo physics engine. To increase the trajectory's complexity, the hopper is thrown up. then rotates and freely falls to the ground (Fig. 3}. We will interpolate the 7-dimensional state that describes the position of the hopper. Both our hidden CSSC and CSSC achieve improved performance from ODE-RNN, especially when observations become sparser, as Tab. | indiicates. The visual result is shown in Fig. 3 where CSSC resemble the GT trajectory most.
4.4 MOVING MNIST
In addition to low now-dimensional data, we further evaluate our method on nigh-dimensional image interpolation. Moving MNIST consists 20-frame video sequences where two handwritten digits are drawn from MNIST and move with arbitrary velocity and direction within the 64 x 64 patches, with potential overlapping and bounce at the boundaries. As a matter of expediency. we use a subset of 10k videos and resize the frames into 32x32. 4 (20%) and 6 (30%) frames out of 20 randomly observed, including the starting and ending frames. We encode the image with ResBlock (He et al.. 2016) into 32-d hidden vector and decode it to pixel space with stack of transpose convolution layers. Since the pixels are only continuous at the semantic level, only hidden CSSC is evaluated with comparison methods. As shown in Tab. the hidden CSSC can further improve ODE-RNN's result, and spline interpolation behaves the worse since it can only interpolate at the pixel space, which is discontinuous through time. The visual result (Fig. 4 shows that the performance gain comes from the smoother movement and the clearer overlapping.
4.5 ABLATION STUDY
The effect of end-to-end training. Apart from our standard end-to-end training of CSSC, two alternative training strategies are pre-hoc and post-hoc CSSN. Pre-hoc CSSC to train standard CSSC but only use the ODE-RNN part when inference. On the contrary. post-hoc CSSC is to train an ODE-RNN without CSSC, but apply CSSC upon the output of ODE-RNN when inference. The
7

---

Under review as a conference paper at ICLR 2021
GT
1
7
0
7
70
70
70
7
7
7
7
7
0
7
0
7
Hidden CSSC
9
?
P
0
0
10
90
0
0
9
9
ODF-RNN
P
P
70
0
9
0
?
Figure 4: The visual result for Moving MNIST. The observation is indicated in white box. The comparison of discontinuity highlighted in red box.
Table 2: The MSE for on MuJoCo set for the study of different training strategies.
10%
30%
50%
ODE-RNN
0.011321
0.001572
0.000388
pre-hoe CSSC
0.053217
0.006131
0.002062
post-hoc CSSC
0.013574
0.000514
0.0001 10
CSSC
0.006097
0.000375
0.000087
comparison of pre-hoc. post-hoc, and standard CSSC is presented in Tab. where standard CSSC behaves the best. The pre-hoc CSSC is the worst because training with CSSC can tolerate the error of the ODE-RNN; thus, inference without CSSC exposes the error of ODE-RNN, and it even performs worse than standard ODE-RNN The post-hoc CSSC can increase the performance of ODE-RNN with simple post-processing when inference. However, such performance gain is not guaranteed when the observation sparse. example, the post-hoe CSSC even decreases the performance of ODE-RNN in 10% observation setting. Standard CSSC has higher performance than ODE-RNN in all our experiments, indicating the importance of end-to-end training.
The effect of a. We study the effect of ranging from 0 to 10000 given different percentages of observation on MuJoCo dataset. The performance CSSC is quite robust to the choice a, shown in Tab. 4 (in Appendix), especially the MSE only fluctuated from 0. 0.000375 to 0.000463 as ranges from to 10000 in 30% observation setting. Interestingly, Fig. (in Appendix) visually compares the interpolation for o, c, and under variant a and indicates higher contributes lower c, thus more dominant of smoothed output ô On the other hand, the smaller a will make less correlated with the ground truth, but the CSSC can always make ô ool-interppolaee curve.
DISCUSSION
Limitations. While the CSSC model interpolates the trajectory that can strictly cross the observation points, such interpolation is not suitable for noisy data whose observations are inaccurate. Moreover, interpolation error bound (Eq. (15) requires the underlying data is fourth-order continuous, which indicates that CSSC is not suitable to interpolate sharply changed data, e.g., step signals.
Future work. The CSSC can not only be applied to ODE-RNN but can smooth any piecewise continuous function. Applying CSSC to other more general models is desirable future work. Also, while interpolation for noisy data is beyond this paper's scope, but hidden CSSC shows the potential to tolerate data noise by capturing the data continuity at the semantic space rather than the observation space, which can be a future direction.
CONCLUSION
We introduce the CSSC that can address the discontinuity issue for ODE-RNN. We have derived the analytical solution for the CSSC and even proved its error bound for the interpolation task, which hard to obtain in pure neural network models. The CSSC combines the modeling ability of deep neural networks (ODE-RNN) and the smoothness advantage of cubic spline interpolation. Our experiments have shown the benefit of such combination. The hidden CSSC extends the smoothness from output space to the hidden semantic space, enabling general format of continuous signals.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Garrett Birkhoff and Arthur Priver. Hermite interpolation errors for derivatives. Journal of Mathe- matics and Physics, 46(1-4):440 447, 1967.
Richard L Burden and Douglas Faires. Numerical analysis, brooks. Cole, Belmont, CA, 1997.
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. In NeurIPS, 2018.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differen- tial equations. In NeurlPS, 2018.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: /406 1078, 2014.
Carl De Boor, Carl De Boor, Etats-Unis Mathématicien, Carl De Boor, and Carl De Boor. A practical guide to splines, volume 27. springer-verlag New York, 1978.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous time series. In NeurlPS, 2019.
modeling of sporadically
Andreas Fagereng and Elin Halvorsen. Imputing consumption from norwegian income and wealth registry data. Journal of Economic and Social Measurement, 42(1)) 2017.
Joseph Futoma, Sanjay Hariharan, and Katherine Heller. Learning detect sepsis with a multitask gaussian process rnn classifier. In Proceedings of the 34th International Conference on Machine Learning- Learning-Volume Volume 70, pp. 1174-1182. JMLR. org, 2017.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural processes. In ICMI, 2018.
Charles A Hall and W Weston Meyer. Optimal error bounds for cubic spline interpolation. Journal of Approximation Theory, 16(2):105-122, 1976.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. arXiv preprint S0Xi::2005.08920 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 14X19:1412.6980, 2014.
Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with rnns: Improved classification of clinical time series. In Machine Learning for Healthcare Conference, pp. 253-270, 2016.
Yonghong Luo, Xiangrui Cai, Ying Zhang, Jun Xu, et al. Multivariate time series imputation with generative adversarial networks. In NeurlPS, 2018.
Michael Moor, Max Horn, Bastian Rieck, Damian Roqueiro, and Karsten Borgwardt. Early recogni- tion of sepsis with gaussian process temporal convolutional networks and dynamic time warping. In Proceedings of the 4th Machine Learning for Healthcare Conference, volume 106, pp. 2-26, 2019.
Michael C Mozer, Denis Kazakov, and Robert Lindsey. Discrete event, continuous time mns. arXiv preprint arXiv: 1710.041 10, 2017.

---

Under review as a conference paper at ICLR 2021
Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M Dai, Nissan Hajaj. Michaela Hardt. Peter J Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, et al. Scalable and accurate deep learning with electronic health records. NPJ Digital Medicine, 1(1):18, 2018.
Yulia Rubanova, Tian Qi Chen, and David K Duvenaud. Latent ordinary differential equations for irreullrly-samplld time series. In NeurIPS, 2019.
Carl Runge. Über empirische funktionen und die interpolation zwischen äquidistanten ordinaten. Zeitschrift für Mathematik und Physik, 46(224-243):20, 1901.
Satya Narayan Shukla and Benjamin Marlin. interpolltioo-prediiiie networks for irregularly sampled time series. In ICLR, 2019.
Qingxiong Tan, Mang Ye, Baoyao Yang. Si-Qi Liu, and Andy Jinhua Ma. Data-gru: Dual-attention time-aware gated recurrent unit for irregular multivariate time series. 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 10X1v:1801.006900, 2018
A
PROOF OF INTERPOLATION
Substitute à with Eq.
S we will have
(x(4)) + o(t)) Ik
(18)
< (1 + 0(T+11) = Ik+1
(19)
á (tx) + +((h)) = én (x + t(t))
(20)
8 (t) + = (tt) + o(t))
(21)
(6()) + = 0
(22)
(x,) + o(T)))
(23)
And we let (tt) = Mr.k 1, n - 1, and ön (1 Mn. We define the first order and second order jump difference of ODE-RNN as
A = o(( ) i((T) ):
(24)
7s ö(t ) ö(tx) )
(25)
With Eq. (21). we have
M (t M + f(tx+1).
(26)
Using constraint Eq.
19
we denote
cx(t)) = I* - -o(tit c
(27)
cx(14+1) -K-I -(((+1) Ex+1'
(28)
Also denote the step size 1k+1 lg. Then applying constraint Eq. piece cubic function expressed as
1 19 21
we have the
Mrt1 Mr (t-tx)3+
MR )2)
1 ck
TM(M++1 + 2M
cx(t)=
)(t +c) (29)
67%
Th
6
Next, we try solve all Mg. We firstly express (( and (x(ttt
Mr
-Ma-1 rk 1 + M 17k +
cx - c
Mr + 2M
(30)
2
-k-1
6
(x(t))
k+1 c
Mr- + 2M
Th
(31)
6
10

---

Under review as a conference paper at ICLR 2021
Applying Eq. 20 have
2Mx+ + - MR-1-
7h
<tt
*x+1 - clt 1, tx
67% 2 -- PK+ITH
Mk+1 6
Tk-1 + Tk
TK- + Tk
Tk-1 + Tk
Th- 1 + Tk
(32)
where c citi 1++1
Tk-1
(33)
Tk- + Th
Tk
(34)
Tk-1 7k
< 11 1++1 < k-
6F
2 1k+17A
d2 = 6
(35)
+ Tk
-
+ Th
Then Ms can be obtained by solving by system of linear equations:
AM d,
(36)
where
2 A1
M1
d1
P22
X2
M2
d2
A
M
d
(37)
Pn-2
2
An-2
Mn-2
dn
Pn-1
Mn-1/
d
And A is non-singular since is striet diagonally dominant matrix, hence guarantee single solution for M'. Hence
M t-
(38)
Now we still need to calculate ö(t) and ö(t).
0g Öh 0g f Öh Öt Ōh
ö(t)
(39)
ii(1) (f((())
0 dh 0 /,
(40)
dt
Th dt
oh
d 0g
0g T
0 g
0g 0f af
a(t) dt
+
f) +
(41)
0h
Th
Öh
Th
B PROOF FOR ERROR BOUND
The following proof are based on the default notation and setting for scalar time series interpolation: given set of n+11 points {= irregularly spaced at time points II to ti ... tr b. the goal is to approximate the underlying ground truth function æ<(), E Ω = (a, b). Let o(l) as the smoothed output where c(t) the compensation defined by Eq. [I]. To simplify the notation, we drop the argument for a function. c.g. x(t) - We firstly introduce several lemmas, then come the the main proof for the interpolation error bound.
ODE-RNN prediction, and ō(t) c(l) o(()
Lemma 4. Hall & Meyer 1976) Let e be any function (II) Ni=n C*ttx11.t tkk) with constraints c044) 0, k U,..., n1 and natural boundary e" e"(a) el)b) 0, then
le S prlle())) 1100
(k O,..,, n: r= 1,2).
(42)
1 Lemma 5. Let = x o, if (*)), f Ct, then e e (I Ni (ta-1::tt))
where P1 24' P2 4.
1.
Proof. Since E (II), and c E N (1 tx), hence we only have to prove that o E
Ni (1(4-11,4)) 4.). Since ot)) in each time period (42-1, (x) is calculated by ODE (Eq. 11. and (2)
can express different orders of the derivative of o
0g T
J
(43)
ah
8
g
0g o
NTS
0h f
(44)
2h
2h
11

---

Under review as a conference paper at ICLR 2021
0f 0og
89g
8g
0f
0g
02
0f 0f
00 - = fT
of +
f
of f
0h 0h ff,
(45)
0h dh?
0h
2h 0h
0h
2h
0 0'g
8
0f 0f
83
05
af 03
8 02 0h Oh
for f +
3
2
f
0h4
0h
0h 0h
0h
0h
8h Th
8 Of 0f
021
0 g
0f
0f
03
0f 0
+ 7h 0h 0h
f
0h
2h
8h
th 2h
0h 0h
0g' T 8'
0
8
0f
Of
0f 0f
0f Of
+
foff
+
o o
f.
0h 0h
0h
0h
0h
0h
Th 2h
8h 0h 0h
(46)
where 0f is the Jacobian matrix since f is a multi-valued function, 8g is the Hessian matrix since o scalar and g is single-valued function. The higher order derivative than second order (Hessian) becomes multi-dimensional matrix which can not be mathmatically expressed for standard matrix production, so we indicate the product of milti-dimmniional matrix and a vector as o, which has higher computing priority then normal matrix production. From Eq. (46). o(4) depends on oh and 8 Hence given g E C and f E C we obtain o E n==1 (tx-1,, 0h
2h
oh
1967) Given any function E (* t + 1), ler be the cubic Hermite interpolation matching v, we have
Lemma 6. (Birkhoff & Priver
(v(()) - -()) 11)
(r= 0, 1),
(47)
with
A1 [[ (1+4 - -]] r!(4 2r)(
if (4,
(48)
This is the error bound for cubic Hermite interpolation.
Given the above lemmas, we are ready to the formal proof for Theorem Proof of Theorem 2
2
Proof. According to Lemma 6 we let v = x - (since E (*t(,, 14-1) according to Lemma let be the cubic Hermite interpolation matching v, then we have
(<x(t) - o(t) - =( s (t)l((z - 110
(r=0,11
(49)
with
A,(t) (: (4++1 - 1))2-r r!(4 - 2r)!74-
if [ta, t,)
(50)
Let ò, u c, then is cubic Hermite interpolation implying
e(1x) = x44)) o(4x) - c((x) = s((x)) - .((+)) = 0,
(51)
and
E(tx) = =(txttt+++)))))>))) - c(tx) = (t++) d(tx) = é(tx).
(52)
Therefore, (f) in [tx, te41) is cubic function with endpoints satisfying can be constructed for each interval [t,, fx+1) Eq. (51) and ( which can be reconstructed
<(() H() + é(lx+1) H2((,,
(53)
where
Hi(t) (t (4) 1++1) 1/22
(54)
Hz(t)= (t tt+1)(t tx /Tē.
(55)
From Lemma and Lemma 5 is bounded as Eq.
42 Combining Eq. (42 and 533
yields:
(r)() !ille (177 (4)
(1))
0, 1),
(56)
12

---

Under review as a conference paper at ICLR 2021
Table 3: The interpolation MSE for toy sinuous We compare the analytical and numerical differentiation of ó and ö for CSSC under different settings, where block means block gradient, drop indicates set as zero, and CSSC is the standard implementation.
at different observation ratio.
Analytical
Numerical
Observation
10%
30%
50%
10%
30%
50%
Block ó, ö
10.601951
0.001408
0.000150
Block ó
0.81 1334
0.000721
0.000142
Block ö
0.406867
0.000348
0.000121
Drop CSSC
0.072519
0.000356
0.000123
0.024656
0.000457
0.000128
0.067881
0.000397
0.000126
which is rewritten
(c(()-((1)) (1 =B, (,(+))((=-
(57)
with B,(1) = p1 (1(1 (1))
Using Triangle inequality, Lemma 6 and Eq. 57 it issues
1z ( (t) |x(()) (t) (t) - ( (t)
[x(( - (0 (0) (1)(()) s (A,(x) + -x( 11)0
-I-
(58)
Let Cx(r) = A,(x) + (z), and an analysis of the optimality (Hall & Meyer 1976 yields
5
1
Co(z) -
Ci(x) -
(59)
384
12'
C COMPUTATIONAL COMPLEXITY
C.I IMPLEMENTATION OF THE INVERSE OF MATRIX
For sake simplicity, our Pytorch implementation adopts torch.inverse inverse to compute A-1 in Eq. (12), which is actually the implementation of the LU composition using partial pivoting with best complexity O(n²). Its complexity is higher than the complexity of tridiagonal matrix algorithm O(n), whose implementation will be left for future work.
C.2 COMPUTATION OF Ó AND ö
As Eq. g indicattes, computing ó and ö requires the Hessian and Jacobian of g and the Jacobian of f. These Jacobians and Hessians will first participate in the inference stage and then are involved the gradient backpropagation. However, the latest Pytoret does not support the computing of Jacobian and Hessian in batch, so the training process will be very slow in practice, rendering the computing cost prohibitively high. Therefore, we propose two ways to eircumvent such issues from both numerical and analytical views.
Pytorch Version 1.6.0, updated in July 2020.
13

---

Under review as a conference paper at ICLR 2021
Numerical Differentiation. The first solution is to use numerical derivative. We approximate the left limitation and right limitation of ö(t) and ö(t) as
ö(t o(t oft At)
(60)
A
o(t) ot 20(t - A) + o(t - -24)
(61)
A
i(()) of + At) - o(t*)
(62)
A
a((T) ol + 2At) 20(t + At) + +((tt)
(63)
A
where A = 0.001. In this way, can avoid computing Jacobian or Hessian, and the computational complexity almost remains the same as ODE-RNN. The last row of Table. 3 shows that the numerical differentiation can maintain the same performance with the analytical solution with 30% and 50% observation. However, when observations become sparser, e.g., 10%, the analytical differentiation will gain better performance.
Analytical Approximation. For analytical solution, the major computing burden is the Hessian matrix; thus, approximate ö with Eq. (65) by simply dropping the Hessian term. The motivation
and rationality is detailed in Appx. C.3
C.3 COMPUTATION REDUCTION FOR ANALYTICAL DERIVATIVE
To further reduce the computation for the analytical derivative of ó and ö, we investigate whether blocking the gradient of ó or ö will affect the interpolation performance. From first three rows in Table. 3 we can see that performance of blocking the gradient of ò is worse than that of blocking ö indicating ò more important than ö in terms of computing the compensation
In light of this, we further drop ö in the Eq. (7). meaning the second order jump difference i'y is zero. According to the performance shown in the fourth row of Table. B ö has minor impact on the compensation e when the observation is dense. We investigate the reason by check how o impact the computation of and find that they are correlated by d2 in Eq. 12). We write de again here for better clarification:
e(tt tw elti ds = 6
6Fg 25xTk-1 - FK+1
(64)
Tk-1 Tk
Tk-1 +
From (Eq. 6 7 and second term of above equation (Eq. as the bridge between ò and ö and dg, indicating that the importance of ó and ö in generating
noticed that tx, Fs serve
compensation e can be examined by estimating the relative significance of rkk Fx appear in dg. We
can denote the relative significance as fraction of terms included 7 and r as 12 +f241Tal < I
[6Az
3 mmax
F Tu Melerrll Without loss of generality, we let max m (WeeT--ll |
then we have 3 61fxl
[Fute-ll It shows that Fx has coefficient of Tk-1 on 2/
the numerator, which is the time interval between two adjacent observations. our experiment, 100 samples is uniformly sampled in 5s, and certain percent of the samples will be selected as observations. In this setting, we have 50% samples as observations, the average тą = 5/50 0.1;
thus 8 can be informally estimated as 8 S 20]pxT. indicating ò is more important than ö in our experiments. As the observation ratio is higher, the тı becomes smaller, hence the relative significance of 1 and F will become even larger. Armed with the above intuition that ő is less important and the fact that he Hessian contribute the major complexity O(W2), drop the term with Hessian and find a better approximation of ö, leading the final approximation of as:
[F&l
0g T T ö
Oh 0h f.
(65)
Such approximation yields descent performance in practice. In addition. because is multi-variable and is a multi-valued function in practice, and such Jacobian needs to run in batch. We implement our Jacobian operation to tackle these difficulties and can run fast.
14

---

Under review as a conference paper at ICLR 2021
Table 4:
The MSE of CSSC for different a MuJoCo test set. It compares the different data
samples.
0
10
100
1000
10000
10%
0.006551
0.011915
0.009691
0.005421
0.006097
0.005886
30%
0.000745
0.000463
0.000426
0.000400
0.000375
0.000422
50%
0.000126
0.000102
0.000074
0.000072
0.000087
0.000057
12-
em0
12-
12-
1110
10-
1.-
1---
06-
06-
46-
92-
-02-
-02-
300
1000
122
10000
10.
1.-
10-
66-
D6-
66-
02
Figure 5: The interpolation MSE CSSC with different a of MuloCo dataset. The curve is the 4-th dimension of the hopper's state.
D IMPLEMENTATION DETAILS
The neural ODE state h has size 15. is 5-layer MLP with hidden state size 300. g is 2-layer MLP with hidden state size 300. RNNCell has hidden state size 100. To obey the condition f e C and g C required by the error bound in Theorem 2 function. The is selected as 1000 based on the ablation study of a in Sec. 4.5 The network is optimized by AdaMax (Kingma Ba 2014) with 51 0.9, B2 0.999, and learning rate 0.02.
we select all the nonlinear function tanh
15


Reviews:
[{"review_id": "wgB98vzl_fk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presented how cubic spline smoothing function was used to compensate the insufficiency in ODE-RNN (ordinary differential equation recurrent neural network) for irregularly sampled sequences.\n\nPros: \nThis paper presented a nice solution to enhance the performance of ODE-RNN with mathematical evidence. Theoretical justification was provided. The solution was evaluated in different tasks.\n\nCons: \nThis solution required the calculation of inverse matrix in a compensation term which would cause a significant increase in training time when the dimension is enlarged.", "extra_scores": {}, "extra_text": {}}, {"review_id": "aJBdyXZ4zex", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper builds on ODE-RNN model that allows to represent a time series as a continuous trajectory. The authors address the limitation of the ODE-RNN model that the trajectory is continuous everywhere except the observation points. They introduce a compensation term based on cubic splines that transforms the output trajectory into a continuous one. This approach is also applied to correct a hidden state trajectory. The authors demonstrate better interpolation properties on sparse time series data.\n\nPros:\n+ The paper targets the main drawback of ODE-RNN that both the output and the hidden state have a \"jump\" at the observation points. The idea of correcting the trajectory or hidden state using cubic splines to make it continuous is novel and interesting.\n\n+ The authors provided the theoretical justification for their method and an interpolation error bound. The authors provided the thorough analysis of the model, including the effect of the numerical differentiation.\n\n+ The authors highlighted the two cases of input data (with the continuity on the input level and on the semantic level) and demonstrated the applicability of CSSC and hidden CSSC in each case. \n\n+ In hidden CSSC, the trajectory is being smoothed out without relying on ground-truth data.\n\nCons:\n- I find the decision to set the outputs exactly to x_k  in standard CSSC to be questionable. Instead of correcting the model predictions, like in hidden CSSC, the model explicitly sets the the predictions to the correct value. First, as authors mention, it restricts the model to the time series with no noise, which has a limited use in practice. Second, with incorrect alpha, the computational capacity is of ODE-RNN is not utilized, and the model reverts to a cubic spline to fit the data. It is visible on figure 5 (alpha <= 10) -- the corrected output \\hat(o)(t) fits the trajectory well despite the poor reconstruction by ODE-RNN.\nIs it possible to do the interpolation in the input space by setting epsilon_k+ = 0; epsilon_k- = o(t_k+) - o(t_k-), similarly to the hidden CSSC? Then the correction c(t) will only play a role of closing the gap between o(t_k+) and o(t_k-). The output of CSSC will remain continuous, but the outputs are no longer restricted to match x_k, improving the expressive power of the model.\n\n- The paper does not provide the error bars for the quantitative results. It is hard to make conclusions about advantages of CSSC and hidden CSSC over other models.\n\n- The proof of interpolation lower bound requires x(t) to have continuous derivatives up to the forth order. It is quite restrictive. I believe that trajectories from Mujoco and moving MNIST already violate this assumption. Can authors hypothesize in which applications we might be able to assume that the inputs belong to C^4?\n\n- Unlike ODE-RNN, the model cannot be used for extrapolation or in the online setting, because the computation of c_k(t) involves the next time point k+1.\n\nOther comments:\n\n-Was the set of subsampled inputs fixed? Was the same set of sampled points used for all the models? In the experiments, only a small fraction of the time points is used as input, and the model performance may change drastically based on the exact set of sampled points. \n\n-I wanted to mentioned that there is a related work by Kidger et al., NeurIPS 2020 \"Neural Controlled Differential Equations for Irregular Time Series\". They use cubic splines on the input data, which allows Neural ODE to query the data at any time point and produce a smooth hidden state trajectory.  The current work achieves a similar goal without modifying the inputs.\n\n-In Table 1 in the first column (Toy data set with 10% observations) Latent ODE actually has the lowest MSE of 0.013768, while CSSC has 0.024656.\n\n-Section 4.5 \"While standard CSSC can always increase the performance upon ODE-RNN\". The increase in performance is not guaranteed. I suggest changing the wording to \"Standard CSSC has higher performance than ODE-RNN in all our experiments\"\n\n\nOverall:\n\nI vote for weak acceptance of the paper. The idea of adding a compensation term resolves the important issue of the ODE-RNN model and is valuable to the ICLR community. I appreciate the idea of hidden CSSC, where only the predicted hidden states were used to produce a smooth trajectory in the latent space. In standard CSSC, my concern is that the predictions at observation times \\hat(t_k) are explicitly set to x_k, making the model inapplicable to noisy data in real-world applications.\n\nI would like to see the error bars for table 1, where the models were trained on the exact same set of subsampled inputs, but with different random seeds. It will help to understand the gap between the cubic splines and CSSC.\n\n============================================\n\nAdded after the author response:\n\nThe authors adequately addressed my questions and concerns. I appreciate that the authors provided the error bars for their experiments and tried out the idea of modifying epsilon_k, but I wish the authors left more time for the discussion. I think the paper is a valuable contribution to the domain of irregular time series. I am increasing my score to 7. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Kg09i2hsr39", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use cubic spline interpolation to smooth out discontinuities outputs and state inferred by ODE-RNNs.\n\nWhile the approach is interesting, it is ill adapted to noisy data (which is a very strong shortcoming IMO) while the experiments are limited to data sets corresponding to applications with limited impact due to their artificial nature.\n\nThe paper has multiple issues which I believe require serious rewriting.\n\n1) The paper has grammatical mistakes including in the abstract and many run-on sentences. The readers could use higher quality writing.\n\n2) The premise that the jumps inside the ODE-RNN are shortcomings or inconsistencies is erroneous. It is expected that their would be jumps with each new observation as the filtration with respect to which the latent state is conditioned changes then with a discontinuity. It would in fact be unexpected to have a continuous posterior. This is for instance what typically happens to stock market stock series any time an earnings report is issued.\n\n3) The experimental section is terse and could really use more diverse datasets. Also comparisons with respect to Gaussian and Neural processes would be welcome. A comparison with a simpler smoothing method such as a convolutions, Fourier and wavelet denoising is also warranted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "125tSkkH_Eq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This work addresses the discontinuity issues caused by jumps in hidden state/output at the arrival of new observations in a ODE-RNN. This problem is tackled by adding a cubic spline smoothing component on top of ODE-RNN to produce smooth and continuous hidden state/outputs. They derive a closed form solution for the cubic spline component based on the output of ODE-RNN and obtain an error bound for 4th order derivable inputs. Although the cubic spline component has no trainable component, this work shows that the gradient can flow through it to ODE-RNN and perform end-to-end training. \n\nPositives:\n1. The paper focus on the task of learning from irregularly sampled data which is important in many domains. \n2. The paper is well written and easy to follow.\n3. Experiments show that the model indeed learns to perform continuous interpolation.\n\nConcerns:\n1. The key concern about the paper is the lack of rigorous experimentation to study the usefulness of the proposed method. The authors mention that this approach would be most useful in continuous time series interpolation for irregularly sampled time series , but there have been no experiments on real world irregularly sampled time series (e.g. PhysioNet). \n2. Another concern I have is with the performance of Latent ODE baseline. The results in Rubanova et al. show that Latent-ODE model almost always outperforms ODE-RNN but that is not the conclusion I get from the results in this paper. This makes me question the validity of implementation/experiments.\n3. ODE-RNN can also be used for extrapolation in addition to interpolation. Is it possible to perform extrapolation using the proposed model? \n\nAdditional Comments:\n1. Could the authors comment on the performance of a Latent ODE model with CSSC as encoder and ODE as decoder? \n2. It would be interesting to see if the improved interpolation leads to improved performance in downstream tasks such as sequence classification.\n2. Missing comparison with kernel based interpolation methods e.g. Shukla and Marlin (2019).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a form of cubic smoothing for use with ODE-RNNs, to remove the jump when new observations occur.  I think this paper's motivation is based on a misunderstanding of what the hidden state of an RNN represents.  Specifically, an RNN hidden state is a belief state, not the estimated state of the system.

I think R2 is right that it's correct for a filter to jump when seeing new data.   It's not a matter of whether the phenomenon being modeled is slow-changing or not.  The filtering output is a belief state, which can change instantaneously even if the true state does not.

The important distinction to make is filtering (conditioning only on previous-in-time data) vs smoothing (conditioning on all data).  The smoothing posterior should generally be smooth if the true state changes slowly.

As R4 notes, all of the tasks are based on interpolation, which is not what the ODE-RNN is trying to do, and the proposed method would make the same predictions as a standard ODE-RNN.  Finally, as R4 notes, "The authors do not provide any experimentation on real-world irregularly sampled time series".
Decision: Reject