Continuous Convolutional Neural Network forNonuniform Time Series

1 INTRODUCTION . Convolutional neural network ( CNN ) , together with recurrent neural network ( RNN ) , is among the most popular deep learning architectures to process time series data . However , both CNN and RNN rest on the assumption that both the input and output data are sampled uniformly . However , many time-series data are event-based and thus not uniform in time , such as stock price ( Gençay et al. , 2001 ) , social media data ( Chang et al. , 2016 ) and health care data ( Johnson et al. , 2016 ) . There are several easy solutions to adapt CNN to accommodate nonuniform time series . The first solution is to directly append the time stamps or time intervals to the input features , which are then fed into a regular CNN ( Zhang et al. , 2017 ) . However , the problem is that , without the uniform sampling assumptions , the application of the regular CNN is ungrounded , and thus the performance is compromised . This is because one major justification of CNN is that the filters/kernels are able to extract useful patterns from input signals . But if the sampling rate varies , the traditional CNN will no longer be able to recognize the same pattern . A second obvious solution is to transform the nonuniform time series to uniform by interpolation , and then feed the transformed signal to a regular CNN . This approach preserves CNN ’ s ability to extract signal patterns despite the nonuniform sampling . However , simple interpolation schemes require preset interpolation kernels , which are not flexible and may not fit the signal or the task well . To sum up , most existing CNN-based remedies for nonuniform time series either can not reasonably capture the signal patterns or are too inflexible to maximize the performance in a data-driven manner . Motivated by these challenges , we propose Continuous CNN ( CCNN ) , a generalization to CNN for nonuniform time series . CCNN estimates the implicit continuous signal by interpolation , yet performs continuous convolution on the continuous signal . As a result , CCNN is capable of capturing the useful patterns in the implicit input signal , which is of nonuniform sampling rate or naturally has an uneven time interval . Furthermore , the interpolation and convolution kernel functions are not preset , but rather learned in an end-to-end manner , so that the interpolation is tailored for each task . Finally , we show that CCNN and CNN are equivalent in terms of representation power under a uniform sampling rate . As shown in section 5 , CCNN can achieve much better performance than the state-of-the-art systems on non-uniform time series data . The proposed CCNN can also be well-combined with the temporal point process ( TPP ) . TPP is a random process of event occurrence , which is capable of modeling nonuniform time intervals . However , most existing TPPs require inflexible preset parameterization . CCNN is able to expand the power of TPP by replacing the modeling of the history contribution with a CCNN module . 2 RELATED WORKS . There are some research efforts of adapting RNN for nonuniform series . Some works ( Pearlmutter , 2008 ; Funahashi & Nakamura , 1993 ; Cauwenberghs , 1996 ) use continuous-time dynamical system methods to design RNN structures . Phased-LSTM ( Neil et al. , 2016 ) and Time-LSTM ( Zhu et al. , 2017 ) introduce a time gate that passes the data at a certain frequency . Similar ideas can be found in Clockwork RNN ( Koutnik et al. , 2014 ) and DilatedRNN ( Chang et al. , 2017 ) . ( Mei & Eisner , 2017 ) and ( Du et al. , 2016 ) explicitly model the sequence as a temporal point process and utilize RNN structure to encode the sequence history . For non-neural network-based approaches , the probabilistic generation process of both events and its timestamps is assumed . ( Liu & Hauskrecht , 2016 ) deals with irregularly sampled time series by direct value interpolation and estimates the model via EM algorithm . ( Wang et al. , 2016 ; Du et al. , 2015 ) base their model on the Hawkes process and estimate via conditional gradient algorithm . The proposed CCNN is well supported by works on spiking neural networks ( SNN ) ( Maass , 1997 ) , which mimic how human brains process information . The inputs to SNNs are spike chains with nonuniform intervals that carry information . An important class SNNs ( Eliasmith & Anderson , 2004 ; Tapson & van Schaik , 2013 ; Tapson et al. , 2013 ) convolves the input with continuous kernel functions , which is similar to the key step of CCNN . However , CCNN differs from SNN in two aspects . First , for SNN , the input information resides in time intervals , not in the inputs values ; the goal of the SNN convolution is to extract time interval information . In contrast , the input information for CCNN resides in input values , not time intervals ; the goal of the continuous convolution is to remove the interference of nonuniform sampling . Second , CCNN learns the kernel functions in a data-driven manner , whereas SNN employs predefined kernel functions . Nonuniform time series processing is related to the task of point set classification , where the input is a set of points nonuniformly distributed in the Rd space ( d = 3 in most cases ) . Several existing methods directly work on the coordinates of the points ( Qi et al. , 2017a ; b ) . Some alternative approaches turn the point sets into graphs by establishing edge connections among nearest neighbors ( Shen et al. , 2017 ; Wang et al. , 2018 ) . The graph approaches utilize distance information to some degree , but the distance information is quantized . CCNN , on the other hand , make full use of the time interval information . 3 THE CCNN ALGORITHM . Our problem is formulated as follows . Given a nonuniform input sequence x ( t1 ) , x ( t2 ) , · · · , x ( tN ) ∈ Xin , where the input time stamps tn ∈ Tin can be distributed nonuniformly , our goal is to design a continuous convolutional layer that can produce output for any arbitrary output time t , y ( t ) . The proposed CCNN solves the problem via two steps : ( 1 ) interpolation to recover the continuous signal x̂ ( t ) ; ( 2 ) continuous convolution on x̂ ( t ) . Furthermore , rather than applying a preset interpolation , CCNN learns the interpolation kernel and the convolution kernel in an end-to-end manner . The following two subsections elaborate on the two steps respectively . The channel dimension and input dimension are set to one for simplification . 3.1 INTERPOLATION . CCNN reconstructs the underlying continuous input signal , x̂ ( t ) , by interpolating among nonuniform input samples . x̂ ( t ) = N∑ i=1 x ( ti ) I ( t− ti ; Tin , Xin ) + ε ( t ; Tin , Xin ) ( 1 ) where the first term is the interpolation term , and I ( · ) is the interpolation kernel ; the second term is the error correction term . For the first term , a form analogous to the Parzen window approach Parzen ( 1962 ) is used . Many interpolation algorithms can be expressed in this form ( refer to Appendix A ) , illustrated in Fig . 6 . Considering the versatility of I ( · ) , the interpolation algorithms representable by Eq . ( 1 ) are vast . The error correction term , ε ( · ) , are assumed to be determined by the input output time stamps and input values , hence its arguments include t , Tin and Xin . 3.2 CONTINUOUS CONVOLUTION . Analogous to a standard CNN layer , after the continuous input is estimated by interpolation , the CCNN layer performs a continuous convolution to produce the final output . y ( t ) = x̂ ( t ) ∗ C ( t ) + b ( 2 ) where ∗ denotes continuous convolution , C ( t ) denotes the convolution kernel , and b denotes bias . Unfortunately , I ( · ) , ε ( · ) and C ( · ) are not individually identifiable . To see this , we combine Eqs . ( 1 ) and ( 2 ) . y ( t ) = N∑ i=1 x ( ti ) [ I ( t− ti ; Tin , Xin ) ∗ C ( t ) ] ︸ ︷︷ ︸ collapsed kernel function + [ ε ( t ; Tin , Xin ) ∗ C ( t ) + b ] ︸ ︷︷ ︸ collapsed bias function = N∑ i=1 x ( ti ) K ( t− ti ; Tin , Xin ) + β ( t ; Tin , Xin ) ( 3 ) whereK ( t ; Tin , Xin ) is the collapsed kernel function , representing the combined effect of interpolation and convolution ; β ( t ; Tin , Xin ) is the collapsed bias function , representing the combined effect of error correction and convolution . Eq . ( 3 ) shows that learning the interpolation and convolution kernels and errors is now simplified into learning the collapsed kernel and bias functions . Once these two functions are learned , the final output can be readily computed using Eq . ( 3 ) . The next section will explain how CCNN is structured to learn these functions in an end-to-end manner . 4 THE CCNN STRUCTURE . Following the discussion in Sec . 3 , a CCNN layer is divided into three parts : the kernel network learning the collapsed kernel function , the bias network learning the collapsed bias function , and the main network producing the final output using Eq . ( 3 ) . 4.1 THE KERNEL NETWORK . The basic idea of the kernel network is to represent the kernel function using a neural network , based on the fact that a neural network can represent any function given enough layers and nodes ( Hornik , 1991 ) . In order to regularize the complexity , a few assumptions on K ( · ) are introduced : Stationarity and Finite Dependency : The dependency of K ( · ) on Tin is relative to the output time t , and is constrained to among the adjacent time stamps , i.e . K ( t− ti ; Tin , Xin ) = K ( { t− ti±k , x ( ti±k ) } k=0 : OK ) ( 4 ) where { t − ti±k , x ( ti±k ) } k=0 : OK denotes the set of t − ti±k and x ( ti±k ) where k runs from 0 to OK , and OK is the order of the kernel network . Notice that the examples in Eqs . ( 14 ) - ( 16 ) and many other interpolation kernels still satisfy this assumption . Finite Kernel Length : The collapsed kernel function has finite length . K ( t− ti ; Tin ) = 0 , ∀|t− ti| > LK ( 5 ) where LK is the kernel length . This assumption implies the interpolation and the convolutional kernels both have finite length . While many interpolation kernels do have finite length ( e.g . Eqs . ( 14 ) and ( 15 ) ) , others do not ( e.g . Eq . ( 16 ) ) . Nevertheless , most infinite-length interpolation kernels , including Eq . ( 16 ) , have tapering tails , and thus truncation on both sides still provides good approximations . Regarding the convolutional kernel , the finite length assumption naturally extends from the standard CNN . Fig . 2 ( a ) shows the kernel network structure , which is a feedforward network . According to Eq . ( 4 ) , the inputs are ( { t−ti±k , x ( ti±k ) } k=0 : OK ) . The output represents the kernel function , which is forced to be zero when |t− ti| > LK . To reduce learning difficulties , the time differences are fed into an optional two-hot encoding layer , which will be discussed later in details .

Summary: The paper proposes a continuous CNN model to accommodate the nonuniform time series data. The model learns the interpolation and the convolution kernel functions in an end-to-end manner, so that it can capture the signal patterns and be flexible. A layer has three networks, which learn a kernel function to represent the combination of interpolation and convolution, a bias function to represent the error correction with convolution, and then produce the output based on them. The authors introduce two assumptions and a two-hot encoding scheme for the input to control the model complexity. The paper also introduces an application of the proposed CCNN by combing with temporal point process. Experiments on simulated data compares the proposed method with some degenerative baselines show the advantage of learning the interpolation and the two-hot encoding configuration. The authors compare the performance on time interval prediction task based on real world dataset to show the model produces a better history embedding for the task.