Cubic Spline Smoothing Compensation for Irregularly Sampled Sequences

The marriage of recurrent neural networks and neural ordinary differential networks ( ODE-RNN ) is effective in modeling irregularly sampled sequences . While ODE produces the smooth hidden states between observation intervals , the RNN will trigger a hidden state jump when a new observation arrives and thus cause the interpolation discontinuity problem . To address this issue , we propose the cubic spline smoothing compensation , which is a stand-alone module upon either the output or the hidden state of ODE-RNN and can be trained end-to-end . We derive its analytical solution and provide its theoretical interpolation error bound . Extensive experiments indicate its merits over both ODE-RNN and cubic spline interpolation . 1 INTRODUCTION . Recurrent neural networks ( RNNs ) are commonly used for modeling regularly sampled sequences ( Cho et al. , 2014 ) . However , the standard RNN can only process discrete series without considering the unequal temporal intervals between sample points , making it fail to model irregularly sampled time series commonly seen in domains , e.g. , healthcare ( Rajkomar et al. , 2018 ) and finance ( Fagereng & Halvorsen , 2017 ) . While some works adapt RNNs to handle such irregular scenarios , they often assume an exponential decay ( either at the output or the hidden state ) during the time interval between observations ( Che et al. , 2018 ; Cao et al. , 2018 ) , which may not always hold . To remove the exponential decay assumption and better model the underlying dynamics , Chen et al . ( 2018 ) proposed to use the neural ordinary differential equation ( ODE ) to model the continuous dynamics of hidden states during the observation intervals . Leveraging a learnable ODE parametrized by a neural network , their method renders higher modeling capability and flexibility . However , an ODE determines the trajectory by its initial state , and it fails to adjust the trajectory according to subsequent observations . A popular way to leverage the subsequent observations is ODE-RNN ( Rubanova et al. , 2019 ; De Brouwer et al. , 2019 ) , which updates the hidden state upon observations using an RNN , and evolves the hidden state using an ODE between observation intervals . While ODE produces smooth hidden states between observation intervals , the RNN will trigger a hidden state jump at the observation point . This inconsistency ( discontinuity ) is hard to reconcile , thus jeopardizing continuous time series modeling , especially for interpolation tasks ( Fig . 1 top-left ) . We propose a Cubic Spline Smoothing Compensation ( CSSC ) module to tackle the challenging discontinuity problem , and it is especially suitable for continuous time series interpolation . Our CSSC employs the cubic spline as a means of compensation for the ODE-RNN to eliminate the jump , as illustrated in Fig . 1 top-right . While the latent ODE ( Rubanova et al. , 2019 ) with an encoder-decoder structure can also produce continuous interpolation , CSSC can further ensure the interpolated curve pass strictly through the observation points . Importantly , we can derive the closed-form solution for CSSC and obtain its interpolation error bound . The error bound suggests two key factors for a good interpolation : the time interval between observations and the performance of ODE-RNN . Furthermore , we propose the hidden CSSC that aims to compensate for the hidden state of ODE-RNN ( Fig . 1 bottom ) , which not only assuage the discontinuity problem but is more efficient when the observations are high-dimensional and only have continuity on the semantic level . We conduct extensive experiments and ablation studies to demonstrate the effectiveness of CSSC and hidden CSSC , and both of them outperform other comparison methods . 2 RELATED WORK . Spline interpolation is a practical way to construct smooth curves between a number of points ( De Boor et al. , 1978 ) , even for unequally spaced points . Cubic spline interpolation leverages the piecewise third order polynomials to avoid the Runge ’ s phenomenon ( Runge , 1901 ) and is applied as a classical way to impute missing data ( Che et al. , 2018 ) . Recent literature focuses on adapting RNNs to model the irregularly sampled time series , given their strong modeling ability . Since standard RNNs can only process discrete series without considering the unequal temporal intervals between sample points , different improvements were proposed . One solution is to augment the input with the observation mask or concatenate it with the time lag ∆t and expect the network to use interval information ∆t in an unconstrained manner ( Lipton et al. , 2016 ; Mozer et al. , 2017 ) . While such a flexible structure can achieve good performance under some circumstances ( Mozer et al. , 2017 ) , a more popular way is to use prior knowledge for missing data imputation . GRU-D ( Che et al. , 2018 ) imputes missing values with the weighted sum of exponential decay of the previous observation and the empirical mean . Shukla & Marlin ( 2019 ) employs the radial basis function kernel to construct an interpolation network . Cao et al . ( 2018 ) let hidden state exponentially decay for non-observed time points and use bi-directional RNN for temporal modeling . Another track is the probabilistic generative model . Due to the ability to model the missing data ’ s uncertainty , Gaussian processes ( GPs ) are adopted for missing data imputing ( Futoma et al. , 2017 ; Tan et al. , 2020 ; Moor et al. , 2019 ) . However , this approach introduced several hyperparameters , such as the covariance function , making it hard to fine-tune in practice . Neural processes ( Garnelo et al. , 2018 ) eliminate such constraints by introducing a global latent variable that represents the whole process . Generative adversarial networks are also adopted for imputing ( Luo et al. , 2018 ) . Recently , neural ODEs ( Chen et al. , 2018 ) utilize a continuous state transfer function parameterized by a neural network to learn the temporal dynamics . Rubanova et al . ( 2019 ) combine the RNN and ODE to reconcile both the new observation and latent state evolution between observations . De Brouwer et al . ( 2019 ) update the ODE with the GRU structure with Bayesian inference at observations . While the ODE produces the smooth hidden states between observation intervals , the RNN will trigger a jump of the hidden state at the observation point , leading to a discontinuous hidden state along the trajectory . This inconsistency ( discontinuity ) is hard to reconcile , thus jeopardizing the modeling of continuous time series , especially for interpolation tasks . The neural CDE ( Kidger et al. , 2020 ) directly apply cubic splines interpolation at the input sequence to make the sparse input continuous and thus produce continuous output . On the contrary , our method tackles this jumping problem by introducing the cubic spline as a compensation for the vanilla ODE-RNN , at either the output or hidden space . 3 METHODS . In this section , we first formalize the irregularly sampled time series interpolation problem ( Sec . 3.1 ) , then introduce the background of ODE-RNN ( Sec . 3.2 ) . Based upon ODE-RNN , we present CSSC and its closed-form solution ( Sec . 3.3 ) , and illustrate the inference and training procedure ( Sec . 3.4 ) . Finally , we provide the interpolation error bound of CSSC ( Sec . 3.5 ) and describe an useful extension of CSSC ( Sec . 3.6 ) . 3.1 PROBLEM DEFINITION . We focus on the interpolation task . Given an unknown underlying function x ( t ) : R→ Rd , t ∈ [ a , b ] , and a set of n + 1 observations { xk|xk = x ( tk ) } nk=0 ∈ Rd sampled from x ( t ) at the irregularly spaced time points Π : a = t0 < t1 < ... < tn = b , the goal is to learn a function F ( t ) : R→ Rd to approximate x , such that F ( tk ) = xk . 3.2 BACKGROUND OF ODE-RNN . ODE-RNN ( Rubanova et al. , 2019 ) achieves the interpolation by applying ODE and RNN interchangeably through a time series , illustrated in top-left of Fig . 1 . The function F on time interval t ∈ [ tk , tk+1 ) is described by a neural ODE with the initial hidden state h ( tk ) : ḣ ( t ) = f ( h ( t ) ) ; ( 1 ) o ( t ) = g ( h ( t ) ) , ( 2 ) where the h ∈ Rm is the hidden embedding of the data , ḣ = dhdt is the temporal derivative of the hidden state , o ∈ Rd is the interpolation output of F ( t ) . Here , f : Rm → Rm and g : Rm → Rd are the transfer function and the output function parameterized by two neural networks , respectively . At the observation time t = tk , the hidden state will be updated by an RNN as : h ( tk ) = RNNCell ( h ( t − k ) , xk ) ; ( 3 ) o ( tk ) = g ( h ( tk ) ) , ( 4 ) where the input x ∈ Rd , t−k and t + k are the left- and right-hand limits of tk . The above formulation has two downsides . The first is the discontinuity problem : while the function described by ODE is right continuous o ( tk ) = o ( t+k ) , the RNN cell in Eq . ( 3 ) renders the hidden state discontinuity h ( t−k ) 6= h ( t + k ) and therefore output discontinuity o ( t − k ) 6= o ( t + k ) . The second is that the model can not guarantee o ( tk ) = xk without explicit constraints . 3.3 CUBIC SPLINE SMOOTHING COMPENSATION . To remedy the two downsides , we propose the module Cubic Spline Smoothing Compensation ( CSSC ) , manifested in the top-right of Fig . 1 . It computes a compensated output ô ( t ) as : ô ( t ) = c ( t ) + o ( t ) , ( 5 ) where o ( t ) is the ODE-RNN output , and the c ( t ) is a compensation composed of piecewise continuous functions . Our key insight is that adding another continuous function to the already piecewise continuous o ( t ) will ensure the global continuity . For simplicity , we set c ( t ) as a piecewise polynomials function and then narrow it to a piecewise cubic function since it is the most commonly used polynomials for interpolation ( Burden & Faires , 1997 ) . As the cubic spline is computed for each dimension of c individually , w.l.o.g. , we will discuss one dimension of the o , c , ô , x and thus denote them as o , c , ô , x , respectively . c ( t ) is composed with pieces as c ( t ) = ∑n−1 k=0 ck ( t ) with each piece ck defined at domain [ tk , tk+1 ) . To guarantee the smoothness , we propose four constraints to ô ( t ) : 1. ô ( t−k ) = ô ( t + k ) = xk , k = 1 , ... , n− 1 , ô ( t0 ) = x0 , ô ( tn ) = xn ( output continuity ) ; 2 . ˙̂o ( t−k ) = ˙̂o ( t + k ) , k = 1 , ... , n− 1 ( first order output continuity ) ; 3 . ¨̂o ( t−k ) = ¨̂o ( t + k ) , k = 1 , ... , n− 1 ( second order output continuity ) ; 4 . ¨̂o ( t0 ) = ¨̂o ( tn ) = 0 ( natural boundary condition ) . The constraint 1 ensures the interpolation curves continuously pass through the observations . Constraint 2 and 3 enforce the first and second-order continuity at the observation points , which usually holds when the underline curve x is smooth . And constraint 4 specifies the natural boundary condition owing to the lack of information of the endpoints ( Burden & Faires , 1997 ) . Given o ( t ) and such four constraints , c ( t ) has unique analytical solution expressed in Theorem 1 . Theorem 1 . Given the first order and second order jump difference of ODE-RNN as ṙk = ȯ ( t + k ) − ȯ ( t − k ) ; ( 6 ) r̈k = ö ( t + k ) − ö ( t − k ) . ( 7 ) where the analytical expression of ȯ and ö can be obtained as ȯ = ∂g ∂h f ; ö = fᵀ ∂2g ∂h2 f + ∂g ∂h ᵀ ∂f ∂h f , ( 8 ) and the error defined as +k = xk − o ( t + k ) ; ( 9 ) −k = xk − o ( t − k ) , ( 10 ) then ck can be uniquely determined as ck ( t ) = Mk+1 + r̈k+1 −Mk 6τk ( t− tk ) 3 + Mk 2 ( t− tk ) 2+ ( −k+1 − + k τk − τk ( Mk+1 + r̈k+1 + 2Mk ) 6 ) ( t− tk ) + +k , ( 11 ) where Mk is obtained as M = A−1d , ( 12 ) A = 2 λ1 µ2 2 λ2 . . . . . . . . . µn−2 2 λn−2 µn−1 2 , M = M1 M2 ... Mn−2 Mn−1 , d = d1 d2 ... dn−2 dn−1 , ( 13 ) τk = tk+1 − tk , µk = τk−1τk−1+τk , λk = τk τk−1+τk , dk = 6 [ t+k , t − k+1 ] − [ t + k−1 , t − k ] τk−1+τk + 6ṙk−2r̈kτk−1−r̈k+1τkτk−1+τk , [ t+k , t − k+1 ] = −k+1− + k τk , M0 = Mn = 0 . The proof for Theorem . 1 is in Appx . A . The c ( t ) is obtained by computing each c ( t ) individually according to Theorem . 1 . Computational Complexity . The major cost is the inverse of A , a tridiagonal matrix , whose inverse can be efficiently computed inO ( n ) complexity with the tridiagonal matrix algorithm ( implementation detailed in Appx . C.1 ) . Another concern is that ȯ and ö needs to compute Jacobian and Hessian in Eq . ( 8 ) . We can circumvent this computing cost by computing the numerical derivative or an empirical substitution , detailed in Appx . C.2 . Model Reduction . Our CSSC can reduce to cubic spline interpolation if setting o in Eq . ( 5 ) as zero . In light of this , we further analyze our model with techniques used for cubic spline interpolation and experimentally show our advantages against it in Sec . 4.2 .

Summary: This work addresses the discontinuity issues caused by jumps in hidden state/output at the arrival of new observations in a ODE-RNN. This problem is tackled by adding a cubic spline smoothing component on top of ODE-RNN to produce smooth and continuous hidden state/outputs. They derive a closed form solution for the cubic spline component based on the output of ODE-RNN and obtain an error bound for 4th order derivable inputs. Although the cubic spline component has no trainable component, this work shows that the gradient can flow through it to ODE-RNN and perform end-to-end training.