Continuous Convolutional Neural Network forNonuniform Time Series

1 INTRODUCTION . Convolutional neural network ( CNN ) , together with recurrent neural network ( RNN ) , is among the most popular deep learning architectures to process time series data . However , both CNN and RNN rest on the assumption that both the input and output data are sampled uniformly . However , many time-series data are event-based and thus not uniform in time , such as stock price ( Gençay et al. , 2001 ) , social media data ( Chang et al. , 2016 ) and health care data ( Johnson et al. , 2016 ) . There are several easy solutions to adapt CNN to accommodate nonuniform time series . The first solution is to directly append the time stamps or time intervals to the input features , which are then fed into a regular CNN ( Zhang et al. , 2017 ) . However , the problem is that , without the uniform sampling assumptions , the application of the regular CNN is ungrounded , and thus the performance is compromised . This is because one major justification of CNN is that the filters/kernels are able to extract useful patterns from input signals . But if the sampling rate varies , the traditional CNN will no longer be able to recognize the same pattern . A second obvious solution is to transform the nonuniform time series to uniform by interpolation , and then feed the transformed signal to a regular CNN . This approach preserves CNN ’ s ability to extract signal patterns despite the nonuniform sampling . However , simple interpolation schemes require preset interpolation kernels , which are not flexible and may not fit the signal or the task well . To sum up , most existing CNN-based remedies for nonuniform time series either can not reasonably capture the signal patterns or are too inflexible to maximize the performance in a data-driven manner . Motivated by these challenges , we propose Continuous CNN ( CCNN ) , a generalization to CNN for nonuniform time series . CCNN estimates the implicit continuous signal by interpolation , yet performs continuous convolution on the continuous signal . As a result , CCNN is capable of capturing the useful patterns in the implicit input signal , which is of nonuniform sampling rate or naturally has an uneven time interval . Furthermore , the interpolation and convolution kernel functions are not preset , but rather learned in an end-to-end manner , so that the interpolation is tailored for each task . Finally , we show that CCNN and CNN are equivalent in terms of representation power under a uniform sampling rate . As shown in section 5 , CCNN can achieve much better performance than the state-of-the-art systems on non-uniform time series data . The proposed CCNN can also be well-combined with the temporal point process ( TPP ) . TPP is a random process of event occurrence , which is capable of modeling nonuniform time intervals . However , most existing TPPs require inflexible preset parameterization . CCNN is able to expand the power of TPP by replacing the modeling of the history contribution with a CCNN module . 2 RELATED WORKS . There are some research efforts of adapting RNN for nonuniform series . Some works ( Pearlmutter , 2008 ; Funahashi & Nakamura , 1993 ; Cauwenberghs , 1996 ) use continuous-time dynamical system methods to design RNN structures . Phased-LSTM ( Neil et al. , 2016 ) and Time-LSTM ( Zhu et al. , 2017 ) introduce a time gate that passes the data at a certain frequency . Similar ideas can be found in Clockwork RNN ( Koutnik et al. , 2014 ) and DilatedRNN ( Chang et al. , 2017 ) . ( Mei & Eisner , 2017 ) and ( Du et al. , 2016 ) explicitly model the sequence as a temporal point process and utilize RNN structure to encode the sequence history . For non-neural network-based approaches , the probabilistic generation process of both events and its timestamps is assumed . ( Liu & Hauskrecht , 2016 ) deals with irregularly sampled time series by direct value interpolation and estimates the model via EM algorithm . ( Wang et al. , 2016 ; Du et al. , 2015 ) base their model on the Hawkes process and estimate via conditional gradient algorithm . The proposed CCNN is well supported by works on spiking neural networks ( SNN ) ( Maass , 1997 ) , which mimic how human brains process information . The inputs to SNNs are spike chains with nonuniform intervals that carry information . An important class SNNs ( Eliasmith & Anderson , 2004 ; Tapson & van Schaik , 2013 ; Tapson et al. , 2013 ) convolves the input with continuous kernel functions , which is similar to the key step of CCNN . However , CCNN differs from SNN in two aspects . First , for SNN , the input information resides in time intervals , not in the inputs values ; the goal of the SNN convolution is to extract time interval information . In contrast , the input information for CCNN resides in input values , not time intervals ; the goal of the continuous convolution is to remove the interference of nonuniform sampling . Second , CCNN learns the kernel functions in a data-driven manner , whereas SNN employs predefined kernel functions . Nonuniform time series processing is related to the task of point set classification , where the input is a set of points nonuniformly distributed in the Rd space ( d = 3 in most cases ) . Several existing methods directly work on the coordinates of the points ( Qi et al. , 2017a ; b ) . Some alternative approaches turn the point sets into graphs by establishing edge connections among nearest neighbors ( Shen et al. , 2017 ; Wang et al. , 2018 ) . The graph approaches utilize distance information to some degree , but the distance information is quantized . CCNN , on the other hand , make full use of the time interval information . 3 THE CCNN ALGORITHM . Our problem is formulated as follows . Given a nonuniform input sequence x ( t1 ) , x ( t2 ) , · · · , x ( tN ) ∈ Xin , where the input time stamps tn ∈ Tin can be distributed nonuniformly , our goal is to design a continuous convolutional layer that can produce output for any arbitrary output time t , y ( t ) . The proposed CCNN solves the problem via two steps : ( 1 ) interpolation to recover the continuous signal x̂ ( t ) ; ( 2 ) continuous convolution on x̂ ( t ) . Furthermore , rather than applying a preset interpolation , CCNN learns the interpolation kernel and the convolution kernel in an end-to-end manner . The following two subsections elaborate on the two steps respectively . The channel dimension and input dimension are set to one for simplification . 3.1 INTERPOLATION . CCNN reconstructs the underlying continuous input signal , x̂ ( t ) , by interpolating among nonuniform input samples . x̂ ( t ) = N∑ i=1 x ( ti ) I ( t− ti ; Tin , Xin ) + ε ( t ; Tin , Xin ) ( 1 ) where the first term is the interpolation term , and I ( · ) is the interpolation kernel ; the second term is the error correction term . For the first term , a form analogous to the Parzen window approach Parzen ( 1962 ) is used . Many interpolation algorithms can be expressed in this form ( refer to Appendix A ) , illustrated in Fig . 6 . Considering the versatility of I ( · ) , the interpolation algorithms representable by Eq . ( 1 ) are vast . The error correction term , ε ( · ) , are assumed to be determined by the input output time stamps and input values , hence its arguments include t , Tin and Xin . 3.2 CONTINUOUS CONVOLUTION . Analogous to a standard CNN layer , after the continuous input is estimated by interpolation , the CCNN layer performs a continuous convolution to produce the final output . y ( t ) = x̂ ( t ) ∗ C ( t ) + b ( 2 ) where ∗ denotes continuous convolution , C ( t ) denotes the convolution kernel , and b denotes bias . Unfortunately , I ( · ) , ε ( · ) and C ( · ) are not individually identifiable . To see this , we combine Eqs . ( 1 ) and ( 2 ) . y ( t ) = N∑ i=1 x ( ti ) [ I ( t− ti ; Tin , Xin ) ∗ C ( t ) ] ︸ ︷︷ ︸ collapsed kernel function + [ ε ( t ; Tin , Xin ) ∗ C ( t ) + b ] ︸ ︷︷ ︸ collapsed bias function = N∑ i=1 x ( ti ) K ( t− ti ; Tin , Xin ) + β ( t ; Tin , Xin ) ( 3 ) whereK ( t ; Tin , Xin ) is the collapsed kernel function , representing the combined effect of interpolation and convolution ; β ( t ; Tin , Xin ) is the collapsed bias function , representing the combined effect of error correction and convolution . Eq . ( 3 ) shows that learning the interpolation and convolution kernels and errors is now simplified into learning the collapsed kernel and bias functions . Once these two functions are learned , the final output can be readily computed using Eq . ( 3 ) . The next section will explain how CCNN is structured to learn these functions in an end-to-end manner . 4 THE CCNN STRUCTURE . Following the discussion in Sec . 3 , a CCNN layer is divided into three parts : the kernel network learning the collapsed kernel function , the bias network learning the collapsed bias function , and the main network producing the final output using Eq . ( 3 ) . 4.1 THE KERNEL NETWORK . The basic idea of the kernel network is to represent the kernel function using a neural network , based on the fact that a neural network can represent any function given enough layers and nodes ( Hornik , 1991 ) . In order to regularize the complexity , a few assumptions on K ( · ) are introduced : Stationarity and Finite Dependency : The dependency of K ( · ) on Tin is relative to the output time t , and is constrained to among the adjacent time stamps , i.e . K ( t− ti ; Tin , Xin ) = K ( { t− ti±k , x ( ti±k ) } k=0 : OK ) ( 4 ) where { t − ti±k , x ( ti±k ) } k=0 : OK denotes the set of t − ti±k and x ( ti±k ) where k runs from 0 to OK , and OK is the order of the kernel network . Notice that the examples in Eqs . ( 14 ) - ( 16 ) and many other interpolation kernels still satisfy this assumption . Finite Kernel Length : The collapsed kernel function has finite length . K ( t− ti ; Tin ) = 0 , ∀|t− ti| > LK ( 5 ) where LK is the kernel length . This assumption implies the interpolation and the convolutional kernels both have finite length . While many interpolation kernels do have finite length ( e.g . Eqs . ( 14 ) and ( 15 ) ) , others do not ( e.g . Eq . ( 16 ) ) . Nevertheless , most infinite-length interpolation kernels , including Eq . ( 16 ) , have tapering tails , and thus truncation on both sides still provides good approximations . Regarding the convolutional kernel , the finite length assumption naturally extends from the standard CNN . Fig . 2 ( a ) shows the kernel network structure , which is a feedforward network . According to Eq . ( 4 ) , the inputs are ( { t−ti±k , x ( ti±k ) } k=0 : OK ) . The output represents the kernel function , which is forced to be zero when |t− ti| > LK . To reduce learning difficulties , the time differences are fed into an optional two-hot encoding layer , which will be discussed later in details .

Summary: A method that was proposed by authors deals with a problem of non-uniform data in time series. One of ways to deal with this problem is interpolate input signal between data points. In signal processing a standard way to interpolate is to apply a convolution with a kernel. This operation, by itself, is a non-trivial, since we lack any information about signal's spectrum (and do not know optimal kernel). Thus, the authors propose to search for a kernel in a form neural network. To this term they also add bias term which is also a neural network.