Cubic Spline Smoothing Compensation for Irregularly Sampled Sequences

The marriage of recurrent neural networks and neural ordinary differential networks ( ODE-RNN ) is effective in modeling irregularly sampled sequences . While ODE produces the smooth hidden states between observation intervals , the RNN will trigger a hidden state jump when a new observation arrives and thus cause the interpolation discontinuity problem . To address this issue , we propose the cubic spline smoothing compensation , which is a stand-alone module upon either the output or the hidden state of ODE-RNN and can be trained end-to-end . We derive its analytical solution and provide its theoretical interpolation error bound . Extensive experiments indicate its merits over both ODE-RNN and cubic spline interpolation . 1 INTRODUCTION . Recurrent neural networks ( RNNs ) are commonly used for modeling regularly sampled sequences ( Cho et al. , 2014 ) . However , the standard RNN can only process discrete series without considering the unequal temporal intervals between sample points , making it fail to model irregularly sampled time series commonly seen in domains , e.g. , healthcare ( Rajkomar et al. , 2018 ) and finance ( Fagereng & Halvorsen , 2017 ) . While some works adapt RNNs to handle such irregular scenarios , they often assume an exponential decay ( either at the output or the hidden state ) during the time interval between observations ( Che et al. , 2018 ; Cao et al. , 2018 ) , which may not always hold . To remove the exponential decay assumption and better model the underlying dynamics , Chen et al . ( 2018 ) proposed to use the neural ordinary differential equation ( ODE ) to model the continuous dynamics of hidden states during the observation intervals . Leveraging a learnable ODE parametrized by a neural network , their method renders higher modeling capability and flexibility . However , an ODE determines the trajectory by its initial state , and it fails to adjust the trajectory according to subsequent observations . A popular way to leverage the subsequent observations is ODE-RNN ( Rubanova et al. , 2019 ; De Brouwer et al. , 2019 ) , which updates the hidden state upon observations using an RNN , and evolves the hidden state using an ODE between observation intervals . While ODE produces smooth hidden states between observation intervals , the RNN will trigger a hidden state jump at the observation point . This inconsistency ( discontinuity ) is hard to reconcile , thus jeopardizing continuous time series modeling , especially for interpolation tasks ( Fig . 1 top-left ) . We propose a Cubic Spline Smoothing Compensation ( CSSC ) module to tackle the challenging discontinuity problem , and it is especially suitable for continuous time series interpolation . Our CSSC employs the cubic spline as a means of compensation for the ODE-RNN to eliminate the jump , as illustrated in Fig . 1 top-right . While the latent ODE ( Rubanova et al. , 2019 ) with an encoder-decoder structure can also produce continuous interpolation , CSSC can further ensure the interpolated curve pass strictly through the observation points . Importantly , we can derive the closed-form solution for CSSC and obtain its interpolation error bound . The error bound suggests two key factors for a good interpolation : the time interval between observations and the performance of ODE-RNN . Furthermore , we propose the hidden CSSC that aims to compensate for the hidden state of ODE-RNN ( Fig . 1 bottom ) , which not only assuage the discontinuity problem but is more efficient when the observations are high-dimensional and only have continuity on the semantic level . We conduct extensive experiments and ablation studies to demonstrate the effectiveness of CSSC and hidden CSSC , and both of them outperform other comparison methods . 2 RELATED WORK . Spline interpolation is a practical way to construct smooth curves between a number of points ( De Boor et al. , 1978 ) , even for unequally spaced points . Cubic spline interpolation leverages the piecewise third order polynomials to avoid the Runge ’ s phenomenon ( Runge , 1901 ) and is applied as a classical way to impute missing data ( Che et al. , 2018 ) . Recent literature focuses on adapting RNNs to model the irregularly sampled time series , given their strong modeling ability . Since standard RNNs can only process discrete series without considering the unequal temporal intervals between sample points , different improvements were proposed . One solution is to augment the input with the observation mask or concatenate it with the time lag ∆t and expect the network to use interval information ∆t in an unconstrained manner ( Lipton et al. , 2016 ; Mozer et al. , 2017 ) . While such a flexible structure can achieve good performance under some circumstances ( Mozer et al. , 2017 ) , a more popular way is to use prior knowledge for missing data imputation . GRU-D ( Che et al. , 2018 ) imputes missing values with the weighted sum of exponential decay of the previous observation and the empirical mean . Shukla & Marlin ( 2019 ) employs the radial basis function kernel to construct an interpolation network . Cao et al . ( 2018 ) let hidden state exponentially decay for non-observed time points and use bi-directional RNN for temporal modeling . Another track is the probabilistic generative model . Due to the ability to model the missing data ’ s uncertainty , Gaussian processes ( GPs ) are adopted for missing data imputing ( Futoma et al. , 2017 ; Tan et al. , 2020 ; Moor et al. , 2019 ) . However , this approach introduced several hyperparameters , such as the covariance function , making it hard to fine-tune in practice . Neural processes ( Garnelo et al. , 2018 ) eliminate such constraints by introducing a global latent variable that represents the whole process . Generative adversarial networks are also adopted for imputing ( Luo et al. , 2018 ) . Recently , neural ODEs ( Chen et al. , 2018 ) utilize a continuous state transfer function parameterized by a neural network to learn the temporal dynamics . Rubanova et al . ( 2019 ) combine the RNN and ODE to reconcile both the new observation and latent state evolution between observations . De Brouwer et al . ( 2019 ) update the ODE with the GRU structure with Bayesian inference at observations . While the ODE produces the smooth hidden states between observation intervals , the RNN will trigger a jump of the hidden state at the observation point , leading to a discontinuous hidden state along the trajectory . This inconsistency ( discontinuity ) is hard to reconcile , thus jeopardizing the modeling of continuous time series , especially for interpolation tasks . The neural CDE ( Kidger et al. , 2020 ) directly apply cubic splines interpolation at the input sequence to make the sparse input continuous and thus produce continuous output . On the contrary , our method tackles this jumping problem by introducing the cubic spline as a compensation for the vanilla ODE-RNN , at either the output or hidden space . 3 METHODS . In this section , we first formalize the irregularly sampled time series interpolation problem ( Sec . 3.1 ) , then introduce the background of ODE-RNN ( Sec . 3.2 ) . Based upon ODE-RNN , we present CSSC and its closed-form solution ( Sec . 3.3 ) , and illustrate the inference and training procedure ( Sec . 3.4 ) . Finally , we provide the interpolation error bound of CSSC ( Sec . 3.5 ) and describe an useful extension of CSSC ( Sec . 3.6 ) . 3.1 PROBLEM DEFINITION . We focus on the interpolation task . Given an unknown underlying function x ( t ) : R→ Rd , t ∈ [ a , b ] , and a set of n + 1 observations { xk|xk = x ( tk ) } nk=0 ∈ Rd sampled from x ( t ) at the irregularly spaced time points Π : a = t0 < t1 < ... < tn = b , the goal is to learn a function F ( t ) : R→ Rd to approximate x , such that F ( tk ) = xk . 3.2 BACKGROUND OF ODE-RNN . ODE-RNN ( Rubanova et al. , 2019 ) achieves the interpolation by applying ODE and RNN interchangeably through a time series , illustrated in top-left of Fig . 1 . The function F on time interval t ∈ [ tk , tk+1 ) is described by a neural ODE with the initial hidden state h ( tk ) : ḣ ( t ) = f ( h ( t ) ) ; ( 1 ) o ( t ) = g ( h ( t ) ) , ( 2 ) where the h ∈ Rm is the hidden embedding of the data , ḣ = dhdt is the temporal derivative of the hidden state , o ∈ Rd is the interpolation output of F ( t ) . Here , f : Rm → Rm and g : Rm → Rd are the transfer function and the output function parameterized by two neural networks , respectively . At the observation time t = tk , the hidden state will be updated by an RNN as : h ( tk ) = RNNCell ( h ( t − k ) , xk ) ; ( 3 ) o ( tk ) = g ( h ( tk ) ) , ( 4 ) where the input x ∈ Rd , t−k and t + k are the left- and right-hand limits of tk . The above formulation has two downsides . The first is the discontinuity problem : while the function described by ODE is right continuous o ( tk ) = o ( t+k ) , the RNN cell in Eq . ( 3 ) renders the hidden state discontinuity h ( t−k ) 6= h ( t + k ) and therefore output discontinuity o ( t − k ) 6= o ( t + k ) . The second is that the model can not guarantee o ( tk ) = xk without explicit constraints . 3.3 CUBIC SPLINE SMOOTHING COMPENSATION . To remedy the two downsides , we propose the module Cubic Spline Smoothing Compensation ( CSSC ) , manifested in the top-right of Fig . 1 . It computes a compensated output ô ( t ) as : ô ( t ) = c ( t ) + o ( t ) , ( 5 ) where o ( t ) is the ODE-RNN output , and the c ( t ) is a compensation composed of piecewise continuous functions . Our key insight is that adding another continuous function to the already piecewise continuous o ( t ) will ensure the global continuity . For simplicity , we set c ( t ) as a piecewise polynomials function and then narrow it to a piecewise cubic function since it is the most commonly used polynomials for interpolation ( Burden & Faires , 1997 ) . As the cubic spline is computed for each dimension of c individually , w.l.o.g. , we will discuss one dimension of the o , c , ô , x and thus denote them as o , c , ô , x , respectively . c ( t ) is composed with pieces as c ( t ) = ∑n−1 k=0 ck ( t ) with each piece ck defined at domain [ tk , tk+1 ) . To guarantee the smoothness , we propose four constraints to ô ( t ) : 1. ô ( t−k ) = ô ( t + k ) = xk , k = 1 , ... , n− 1 , ô ( t0 ) = x0 , ô ( tn ) = xn ( output continuity ) ; 2 . ˙̂o ( t−k ) = ˙̂o ( t + k ) , k = 1 , ... , n− 1 ( first order output continuity ) ; 3 . ¨̂o ( t−k ) = ¨̂o ( t + k ) , k = 1 , ... , n− 1 ( second order output continuity ) ; 4 . ¨̂o ( t0 ) = ¨̂o ( tn ) = 0 ( natural boundary condition ) . The constraint 1 ensures the interpolation curves continuously pass through the observations . Constraint 2 and 3 enforce the first and second-order continuity at the observation points , which usually holds when the underline curve x is smooth . And constraint 4 specifies the natural boundary condition owing to the lack of information of the endpoints ( Burden & Faires , 1997 ) . Given o ( t ) and such four constraints , c ( t ) has unique analytical solution expressed in Theorem 1 . Theorem 1 . Given the first order and second order jump difference of ODE-RNN as ṙk = ȯ ( t + k ) − ȯ ( t − k ) ; ( 6 ) r̈k = ö ( t + k ) − ö ( t − k ) . ( 7 ) where the analytical expression of ȯ and ö can be obtained as ȯ = ∂g ∂h f ; ö = fᵀ ∂2g ∂h2 f + ∂g ∂h ᵀ ∂f ∂h f , ( 8 ) and the error defined as +k = xk − o ( t + k ) ; ( 9 ) −k = xk − o ( t − k ) , ( 10 ) then ck can be uniquely determined as ck ( t ) = Mk+1 + r̈k+1 −Mk 6τk ( t− tk ) 3 + Mk 2 ( t− tk ) 2+ ( −k+1 − + k τk − τk ( Mk+1 + r̈k+1 + 2Mk ) 6 ) ( t− tk ) + +k , ( 11 ) where Mk is obtained as M = A−1d , ( 12 ) A = 2 λ1 µ2 2 λ2 . . . . . . . . . µn−2 2 λn−2 µn−1 2 , M = M1 M2 ... Mn−2 Mn−1 , d = d1 d2 ... dn−2 dn−1 , ( 13 ) τk = tk+1 − tk , µk = τk−1τk−1+τk , λk = τk τk−1+τk , dk = 6 [ t+k , t − k+1 ] − [ t + k−1 , t − k ] τk−1+τk + 6ṙk−2r̈kτk−1−r̈k+1τkτk−1+τk , [ t+k , t − k+1 ] = −k+1− + k τk , M0 = Mn = 0 . The proof for Theorem . 1 is in Appx . A . The c ( t ) is obtained by computing each c ( t ) individually according to Theorem . 1 . Computational Complexity . The major cost is the inverse of A , a tridiagonal matrix , whose inverse can be efficiently computed inO ( n ) complexity with the tridiagonal matrix algorithm ( implementation detailed in Appx . C.1 ) . Another concern is that ȯ and ö needs to compute Jacobian and Hessian in Eq . ( 8 ) . We can circumvent this computing cost by computing the numerical derivative or an empirical substitution , detailed in Appx . C.2 . Model Reduction . Our CSSC can reduce to cubic spline interpolation if setting o in Eq . ( 5 ) as zero . In light of this , we further analyze our model with techniques used for cubic spline interpolation and experimentally show our advantages against it in Sec . 4.2 .

Summary: The paper builds on ODE-RNN model that allows to represent a time series as a continuous trajectory. The authors address the limitation of the ODE-RNN model that the trajectory is continuous everywhere except the observation points. They introduce a compensation term based on cubic splines that transforms the output trajectory into a continuous one. This approach is also applied to correct a hidden state trajectory. The authors demonstrate better interpolation properties on sparse time series data.