Continuous Convolutional Neural Network forNonuniform Time Series
Authors: Hui Shi
Yang Zhang
Hao Wu
Shiyu Chang
Kaizhi Qian
Mark Hasegawa-Johnson
Jishen Zhao
Venue:  (2020)

Abstract: Convolutional neural network (CNN) for time series data implicitly assumes that the data are uniformly sampled, whereas many event-based and multi-modal data are nonuniform or have heterogeneous sampling rates. Directly applying regularCNN to nonuniform time series is ungrounded, because it is unable to recognize and extract common patterns from the nonuniform input signals. Converting the nonuniform time series to uniform ones by interpolation preserves the pattern extraction capability of CNN, but the interpolation kernels are often preset and may be unsuitable for the data or tasks. In this paper, we propose the ContinuousCNN (CCNN), which estimates the inherent continuous inputs by interpolation, and performs continuous convolution on the continuous input. The interpolation and convolution kernels are learned in an end-to-end manner, and are able to learn useful patterns despite the nonuniform sampling rate. Besides, CCNN is a strict generalization to CNN. Results of several experiments verify that CCNN achieves abetter performance on nonuniform data, and learns meaningful continuous kernels

Under review as a conference paper at ICLR 2020
CONTINUOUS CONVOLUTIONAL NEURAL NETWORK FOR NONUNIFORM TIME SERIES
Anonymous authors Paper under double-blind review
ABSTRACT
Convolutional neural network (CNN) for time series data implicitly assumes that the data are uniformly sampled, whereas many event-based and multi-modal data are nonuniform or have heterogeneous sampling rates. Directly applying regular CNN to nonuniform time series is ungrounded because it is unable to recognize and extract common patterns from the nonuniform input signals. Converting the nonuniform time series to uniform ones by interpolation preserves the pattern extraction capability CNN, but the interpolation kernels are often preset and may be unsuitable for the data tasks. In this paper, we propose the Continuous CNN (CCNN), which estimates the inherent continuous inputs by interpolation, and performs continuous convolution and convolution kernels are learned in an end-to-ene manner, and are able to learn useful patterns despite the nonuniform sampling rate. Besides, CCNN is strict generalization to CNN. Results of several experiments verify that CCNN achieves a better performance on nonuniform data, and learns meaningful continuous kernels.
the continuous input. The interpolation
INTRODUCTION
Convolutional neural network (CNN), together with recurrent neural network (RNN), is among the most popular deep learning architectures to process time series data. However, both CNN and RNN rest on the assumption that both the input and output data are sampled uniformly. However, many time-series data are event-based and thus not uniform in time, such as stock price (Gençay et al. 200T], social media data (Chang et al.) 2016) and health care data (Johnson et al. 2016).
There are several easy solutions to adapt CNN to accommodate nonuniform time series. The first solution is to directly append the time stamps or time intervals to the input features, which are then fed into regular CNN (Zhang al. 2017). However, the problem is that, without the uniform sampling assumptions, the application of the regular CNN is ungrounded, and thus the performance is compromised. This is because one major justification of CNN is that the filters/kernels are able to extract useful patterns from input signals. But if the sampling rate varies, the traditional CNN will no longer be able to recognize the same pattern.
second obvious solution to transform the nonuniform time series to uniform by interpolation, and then feed the transformed signal regular CNN. This approach preserves CNN's ability extract signal patterns despite the nonuniform sampling. However, simple interpolation schemes require preset interpolation kernels, which are not flexible and may not fit the signal or the task well. To sum up, most existing CNN-based remedies for nonuniform time series either cannot reasonably capture the signal patterns or are too inflexible to maximize the performance
data-driver manner.
Motivated by these challenges, we propose Continuous CNN (CCNN), generalization to CNN for nonuniform time series. CCNN estimates the implicit continuous signal by interpolation, yet performs continuous convolution on the continuous signal. As result, CCNN is capable of capturing the useful patterns in the implicit input signal, which is of nonuniform sampling rate or naturally has uneven time interval. Furthermore, the interpolation and convolution kernel functions are not preset, but rather learned in an end-to-end manner, so that the interpolation is tailored for each task. Finally, show that CCNN and CNN are equivalent in terms of representation power under uniform sampling rate. As shown in section [5] CCNN can achieve much better performance than the fate-of-the-ar systems on non-uniform time series data.

---

Under review as a conference paper at ICLR 2020
Prediction
Normal Convolution
(Kernel length 4)
v(C))
yt
vic
Y0
yatal
ytra)
vtead
CCNN outpul
(uniform)
Continuous
Convolution
(Kernel length Le)
Input signal (non-umiform
x(5) x(x) x
x(t)
(( xttal
x(Ly)
x(te)
x(g1)
te's Lyc (a
to Dic
Figure 1: CCNN Structure for predicting future events. Given uniform-inteen time sequence {Ei CCNN layer performs both interpolating non-uniformly sampled signal sequence {a(tt)) (á(t()) and (onvolution(uu(t))) Now that {u(t))) uniformly resampled, normal convolution layer can be applicd.
The proposed CCNN can also be well-combined with the temporal point process (TPP). TPP is random process of event occurrence, which is capable of modeling nonuniform time intervals. However, most existing TPPs require inflexible preset parameterization. CCNN is able to expand the power of TPP by replacing the modeling of the history contribution with CCNN module.
RELATED WORKS
There are research efforts of adapting RNN for nonuniform series. Some works (Pearlmutter 2008 Funahashi & Nakamura, 1993 Cauweeberrghs 1996) use continuous-time dynamical system methods design RNN structures. Phased-LSTM (Neil et al. 2016) and Time-LSTM (Zhu et al.) 2017) introduce time gate that passes the data at a certain frequency. Similar ideas can be found in Clockwork RNN (Koutnik al., 2014) and DilatedRNN (Chang al. 2017). (Mei & Eisner 2017) and (Du et al. 2016) explicitly model the sequence as a temporal point process and utilize RNN structure to encode the sequence history.
For non-neural network-basse upproaches, the probabilistic generation process of both events and its timestamps is assumed. (Liu Hauskrecht] 2016) deals with irregularly sampled time series by direct value interpolation and estimates the model via EM algorithm. (Wang et al. 2016 6 Du et al.) 2013) base their model on the Hawkes process and estimate via conditional gradient algorithm.
The proposed CCNN is well supported by works spiking neural networks (SNN) (Maass) 1997). which mimic how human brains process information. The inputs to SNNs are spike chains with nonuniform intervals that carry information. An important class SNNs (Eliasmith & Anderson 2004 Tapson & van Schaik 2013 Tapson et al.) (201)) convolves the input with continuous kernel functions, which is similar to the key step of CCNN. However, CCNN differs from SNN in two aspects. First, for SNN, the input information resides in time intervals, not in the inputs values; the goal of the SNN convolution is extract time interval information. In contrast, the input information for CCNN resides in input values, not time intervals; the goal of the continuous convolution is to remove the interference nonuniform sampling. Second, CCNN learns the kernel functions in a data-driven manner, whereas SNN employs predefined kernel functions.
Nonuniform time series processing related the task point classification, where the input is set of points nonuniformly distributed in the Id space (d 3 in most cases). Several existing methods directly work on the coordinates of the points (Qi et al. 2017a b). Some alternative approaches turn the point sets into graphs by establishing edge connections among nearest neighbors (Shen et al. 2017; Wang et al. 2018). The graph approaches utilize distance information to some degree, but the distance information is quantized. CCNN, the other hand, make full use of the time interval information.
3 THE CCNN ALGORITHM
Our problem is formulated as follows. Given a nonuniform input sequence x(t)), x(tx) e Xin. where the input time stamps t, Im can be distributed nonuniformly, goal is design contimuous convolutional layer that can produce output for any arbilrary output time I, y()).
The proposed CCNN solves the problem via steps:
2

---

Under review as a conference paper at ICLR 2020
(I) interpolation to recover the continuous signal #(t);
(2) continuous convolution on (t)).
Furthermore, rather than applying a preset interpolation, CCNN learns the interpolation kernel and the convolution kernel in an end-to-end manner. The following two subsections elaborate on the two steps respectively. The channel dimension and input dimension are set to one for simplification.
3.1
INTERPOLATION
CCNN reconstructs the underlying continuous input signal, att), by interpolating among nonuniform input samples.
a(1) - Σ (l)) 4;; Jin xin) Fin
where the first is the interpolation term, and (()) is the interpolation kernel; the second is the error correction term. For the first term, a form analogous to the Parzen window approach Parzen (1962) is used. Many interpolation algorithms can be expressed in this form (refer to Appendix A), illustrated in Fig. 6 Considering the versatility of I(-), the interpolation algorithms representable by Eq. ① are vast. The eтrоr correction term, E(-), are assumed to be determined by the input output time stamps and input values, hence its arguments include I, Fin and Xin.
3.2 CONTINUOUS CONVOLUTION
Analogous to a standard CNN layer, after the continuous input is estimated by interpolation, the CCNN layer performs continuous convolution to produce the final output.
s(t) = (z()) C(t) + b
(2)
where denotes continuous convolution, C(t) denotes the convolution kernel, and b denotes bias. Unfortunately, I(-), E-) and C(-) are not individually identifiable. To see this, we combine Eqs. and 2
v(t) Σ (t,)) [[(t t;; Jm. X,n)) C(t) + [e(t; Imn Xin) C(t) b
collapsed kernel function
collapsed bias function
(3)
Σ
Tn. B(t Im. Xin)
where K (1;; Cin) is the collapsed kernel function, representing the combined effect of interpolation and convolution; (t; Tm..(nn)) ) is the collapsed bias function, representing the combined effect of error correction and convolution.
Eq. shows that learning the interpolation and convolution kernels and errors is now simplified into learning the collapsed kernel and bias functions. Once these two functions are learned, the final output can be readily computed using Eq. (3). The next section will explain how CCNN is structured to learn these functions an end-to-end manner.
THE CCNN STRUCTURE
Following the discussion in Sec. 3 CCNN layer is divided into three parts: the kernel network learning the collapsed kernel function, the bias network learning the collapsed bias function, and the main network producing the final output using Eq. (3).
4.1 THE KERNEL NETWORK
The basic idea of the kernel network is represent the kernel function using neural network, based on the fact that a neural network can represent any function given enough layers and nodes (Homik 1991). In order regularize the complexity, few assumptions on K(.) introduced:
3

---

Under review as a conference paper at ICLR 2020
Kit
p xv ml
Feodforward Network
Feodforward Network
X()
Concatenate
Concatenate
Ptr: I w &
Sum
Twa-hot Encoding
bias network outpur /
Two-hot Encoding
Inner Product
L <. Al,
W
RV Ev
xitel
30-0,1
A6A
Inpurt sequence X
kernel network output KD
ixpur sequence Ta
Enpull sequence X
5p-
5.10
(c) Compute y(t) by collapsed interpolation and convolution. K(t (4) is the shorthand
1-1:0- time differonce Ar
1r
time difference
for K(t - Fe,,
(a) Kernel network.
(b) Bias network.
Figure 2: CCNN structure.
Stationarity and Finite Dependency: The dependency of K(.) Ii is relative to the output time t, and is constrained to among the adjacent time stamps, i.e.
K(t - t;: Xm. = K(ft ts x(th:=))
(4)
where {t - LiikR .rllitt ( k=0000x denotes the set of t lin and a((4t)) where k runs from 0 to Ok, and Ox is the order the kernel network. Notice that the examples in Eqs. (14) (T6) and many other interpolation kernels still satisfy this assumption.
Finite Kernel Length: The collapsed kernel function has finite length.
K(L - L; Fmm = 0, 11 - 4 > Lx
(5)
where 4x is the kernel length. This assumption implies the interpolation and the convolutional kernels both have finite length. While many interpolation kernels do have finite length (e.g. Eqs. {4) and
(15)). others do not (e.g. Eq. (16). Nevertheless, most infinite-length interpolation kernels, including
Eq. (16), have tapering tails, and thus truncation on both sides still provides good approximations. Regarding the convolutional kernel, the finite length assumption naturally extends from the standard
CNN.
Fig. 2 shows the kernel network structure, which is feedforward network. According to Eq. (4. the inputs are ((t - f .r (f==)))===00o)) ). The output represents the kernel function, which forced to be zero when it t, Lg. To reduce learning difficulties, the time differences fed into an optional two-hot encoding layer, which will be discussed later in details.
4.2 THE BIAS NETWORK
For the bias network, a similar stationarity and finite dependency assumption is applied as follows.
Fm. Xin) = B({ t #hs =(ty==)))==000
(6)
where tz- is the closest input time stamp output time t:
ty- argmin Ity t| 63.
and On denotes the order of the bias network. The only difference from Eq. is that the closest input time stamp. 1j=, is chosen as a reference on which the time difference and the adjacent input time stamps are defined, because the major argument of the bias function is the output time itself, t, instead of the input-output time difference f - t. Fig. bb shows the bias network, which is also a feedforward network.
4.3 CAUSAL SETTING
For causal tasks, current output should not depend on future input, and therefore the t. terms that are greater than 0, well the corresponding 2(tx+)), are removed from Eq. 4 Similarly, t - as that are greater than 0, as well as the corresponding o((y==*) are removed from Eq. (6)
Also, the condition bound in Eq. (5) is replaced with - 1, > Tag or L < 0.
4

---

Under review as a conference paper at ICLR 2020
1.0
1.0
8.4
8.
0.00
0.00
k:2
k-1
K
K
k
k-2
k
11 2
11x
Th
TG
11
Figure 3: Two-hot encoding. Each cross the 1-D axis denotes value of At. The stem plot above shows its twa-hot vector, Assuming = 5, left plot shows when At TT- 0.46, the twa-hot encoding of is [0, 0.6, 0.4, O, 0j. The right plot shows when At Tk-1, the encoding is [0, 1, 0,
4.4
TWO-HOT ENCODING
The kernel and bias functions can be complicated functions of the input times, so model complexity and convergence can be serious concerns. Therefore, we introduce two-hot encoding scheme for the input times, which is an extension the one-hot scheme, but which does not lose information.
Denote the time difference value to be encoded as At. Similar to one-hot, the two-hot encoding scheme partitions the range of At into D intervals, whose edges are denoted as T1 "2.-.. ... wd- However, rather than having length-D D 1 vector representing the intervals, two-hot introduces length-D vector representing the edges. When A/ falls an interval, the two elements corresponding to its two edges are lit. Formally, denote the encoded vector as g. and suppose A/ falls in interval [TTy Tk+1). Then
At
At Te
9t = O, 1A {ki 1]
(8)
where 5 = Ty Tk-1 denotes the interval width (all the intervals are set to have equal width); 9 denotes the k-th element of g. Fig. gives intuitive visualization of the encoding process.
As an example explanation why two-hot helps, it can be easily shown that a one-layer feed-forwart network only learn linear function (a straight line) without any encoding, but piecewise constant function with one-hot encoding, and yet a piecewise linear function with two-hot encoding.
4.5 COMBINING WITH TEMPORAL POINT PROCESSES
For tasks like predicting the time interval till the next event, the output of CCNN will be the predicted probability distribution of the time interval, which requires good probabilistic model characterizing the likelihood of these intervals. Temporal point process (TPP) a popular and general model for the timestamps of the random processes (xxtt),, & whose time intervals are nonuniform. It turns out that CCNN can be well combined with TPPs in modeling the time interval prediction task, in a similar way to Du et al. (2016).
TPP is parameterized by X (t), which depicts the rate of the probability of the event occurrence. Formally
X (t)dt Pr Event happens in [t, dt)
Utz(s))
It can be shown that the probability density function (PDF) of an event happening at time i conditional on the history of the events 1 {x((t),, can be expressed as
f (t) = (t) exp (-f.__ x(()))))
(10)
Rather than applying some preset functional form for (1) as in conventional TPPs, we propose use a CCNN to model X (t) as follows. First, we pass the historical time series to CCNN to learn a history embedding
U (ltt))))))) J51-1
h CCNN
(11)
where CCNN (-) is just a functional abstraction of CCNN. Then (+ is obtained by combining the history information and the current time information as follows
X (1) exp(eh;-1 + w(l 4-1) b),
(12)
5

---

Under review as a conference paper at ICLR 2020
Kemel Weight KlAt)
Alg.
Sine
MG
Lorenz
1-
CNN
46.0 (8.22)
12.8 (3.92)
9.90 (3.33)
Kendl Weight
CNNT
20.2 (7.65)
3.50 (1.29)
5.97 (2.41)
0
CN
8,44 (4.58)
3.00 (1.21)
8.37 (3.24)
2:5
ICNN-L
1.13 (0.87)
0.97 (0.53)
5,81 (2.78)
ICNN-Q
0.75 (0.65)
0.83 (0.46)
5.08 (2.59)
(3) AT (Petiod=4)
dt (Peroo-)))
ICNN-C
0.72 (0.83)
(0.42)
4.22 (2.27)
Kamal Weghn K(az)
Kemal Weight Kldt)
ICNN-P
20.5(6.43)
1.95(0.79)
8,50(3.32)
ICNN-S
17.2(5.57)
3.51(1.36)
8.20(3.31)
0
RNNT
36.1(12.9)
8.15(3.32)
13.4(3.95)
RNNT-th
19,5(6.48)
8,48(3.11)
13.9
T72 3.0
-]
T/2 m 3.5
CCNN
0.88 (0.61)
2.46 (0.89)
3,93 (1.73)
3 (Period-))
CCNN-th
0.42 (0.36)
0.53 (0.97)
3.25 (1.67)
(3) (Period-5)
Table Mean squared enror prediction on simulated data (x10 3).
Figure The learned continuous kernel function on the sine, as functions of AI 4.
where v, and b are trainable parameters. Combining Eqs. (T)) and we can obtain closed-form expression for f (t)
(1) exp (hh- + wit 12-1) + bb 2 w
(exp(eh, + exp(uhy-11100 wi (((-
(13)
By maximizing this likelihood the training data, estimate the conditional distribution of the time intervals. To obtain a point estimate of the time interval till the next event, we compute the expectation under Eq. (13) numerically.
Du et al. (2016) also applies the same approach, but the history embedding h;-1 is computing by regular RNN. CCNN, with its improved processing of nonuniform data, is expected produce better history embedding, and thereby a better estimate of the time intervals.
4.6
SUMMARY AND GENERALIZATION
Fig. illustrates the structure of a CCNN layer. To sum up. the kernel network and bias network learn the continuous kernel and bias functions of I, In and . The main network applies these functions to produce the output according to Eq. 33. The hyperparameters include Ox (Eq. (4 ), Lx
(Eq. ), OB (Eq.66) and (Eq. (8).
It is worth highlighting that CCNN not only accommodates arbitrary input timestamps, it can also produce output at any output timestamps, by simply adjusting the value of Eqs. (3) and (o). So CCNN layer can accept imput at set of timestamps, and produces output at different set of timestamps, which very useful for resampling, interpolation, and continuous sequence prediction.
When the inputs stt;) and output yt) need to be multidimensional, according to Eq. 33. K(-) and 3(.) become vectors or matrices with matching dimensions. Therefore, we simply need adapt the output of the kernel and the bias networks from scalars vectors vectorized matrices. Also, multi-layer CCNN can be constructed by stacking individual CCNN layers, with the input timestamps of layer matching the output timestamps of its previous layer.
EVALUATION
In this section, CCNN is evaluated of prediction tasks both simulated and real-world data.
5.1 PREDICTING SIGNAL VALUE ON SIMULATED DATA
The prediction task predicts the next sample z(tN41), given the previous nonuniform samples <(41) x!xx).
Datasets The synthetic datasets generated by unevenly sampling from three standard time series: Sine, Mackey-Glasx(MG) and Lorenz. The details of the time series are introduced Appendix C.
Baselines The following algorithms compared:
CCNN: The proposed algorithm. The first layer is a CCNN layer which takes both the sampling time intervals and the signal sequence. After the CCNN layer, the sequence is resampled onto

---

Under review as a conference paper at ICLR 2020
uniform time interval. For predicting future label, signal value, and interval, the CCNN is configured
using only past value, i.e. CCNN kernel has non-zero value only when t' LK < t, < t', shown
in Fig. DI The time information is either two-hot encoded (Adams et al. 2010 (CCNN-th) or not encoded (
CNN: data are directly fed into regular CNN, with no special handling of nonuniform sampling-
CNNT: The sampling time intervals are appended to the input data, which are fed to a regular CNN. The time information is either two-hot encoded (CNNT-th)), or not encoded (CNNT).
ICNN: data are interpolated to be uniform before being fed to a regular CNN. Piecewise Con- stant (ICNN-P), linear (ICNN-L), quadratic (ICNN-Q), cubic spline (ICNN-C) and sinc (ICNN-S) interpolation algorithms are implemented.
RNNT: the sampling time intervals are appended to the input data, then are fed into vanilla RNN. The time information is either two-hot encoded (RNNT-thh) or not encoded (RNNT).
All the networks have two layers with ReLU activations in the hidden layers and activations in the output layer. For CNN, ICNN, and CNNT, the convolution kernel length of each layer is set to 7. For ICNN, the input signal is interpolated at timestamps f K. k 1,... 13 to form a uniform sequence before feeding into two-layers regular CNN. For CCNN, the output time stamps of the first layer are NN+1 - k. k 1, ... 13. The kernel length Lg- = Since its input uniform, the second layer of CCNN is a regular convolutional layer, with kernel length 7. These configurations ensure that all the neural networks have the same expected receptive field size of 13.
The rest of the hyperparameters are set such that all the networks have comparable number of parameters, as in Appendix C.2, All the networks are trained with Adam optimizer Kingma Ba (2014) and mean squared error (MSE) loss. The training batch size is 20. The number of training steps is determined by validation. The validation set size is 10,000.
Results and Analysis Table lists the MSEs. There are three observations. First, CCNN-th out- performs the other baselines in terms of prediction accuracy. Notice that the number of convolution channels of CCNN are significantly smaller than most of the other baselines, in order to match the number of parameters. Nevertheless, the advantage in properly dealing with the nonuniform sampling still offsets the reduction in channels in most tasks. Second, interpolation methods (ICNNs and CCNN) generally outperform the other baselines, particularly CNNT. This again shows that interpolation is more reasonable for dealing with nonuniform time series than simply appending the time intervals. Furthermore, preset interpolation algorithms (ICNNs) can rarely match CCNN that has the flexibility to learn its own interpolation kernel. Third, two-hot encoding usually improves perfor- mance. Again, there are fewer channels with two-hot encoding in order to match model complexity. but the advantage of two-hot encoding still stands out.
Kernel Analysis In order to visualize the learned continuous kernels K - f,; Tm: I,n), we set the CCNN network has the same configuration except CCNN filter number is 1, and is trained separately with nonuniformly sampled (A = 1) sine signals with T = 4, 5, 6, 7. The learned continuous kernel function is quite interpretable. Each kernel is a sine-like function with estimated period equaling the underlying signal period shown in Fig. 4
5.2 PREDICTING TIME INTERVALS TO NEXT EVENT
We then evaluate CCNN on real-world data to to predict the time intervals to the next event.
Datasets and Configurations Four time series datasets with nonuniform time intervals are intro- duced, i.e. NYSE, Stackoverflov@] MIMIC (Johnson et al. (2016) and Retweet ((Zhao et al. 20155). Detailed information are provided in Appendix C..3 The input sequence is the timestamps and the one-hot encoded types of a series of events. The task to predict the time interval until the next event of specific type, given the previous events. As mentioned in Sec. 4.5 following the design in (Du) ct al., 2016), the input sequence is assumed to be generated via an underlying marked TPP, where the time stamps follow TPP, and the marker, i.e. the type of the event, is generated from multinomial distribution conditioned on history. The output of the networks is a condition intensity function X (t)
'erreatter, in tables, the numbers in parentheses show standard deviation. https//acchheeoogglliiiiittehhhe
7

---

Under review as a conference paper at ICLR 2020
10
Intevel
bogen Interal
Evenr Intertal
Eigni
1644
KMASI
RMMSE
11.24
14-
9.3
56-
12
13
05
48
00
Mudcl
Mudele
Mideis MIMIC
Modck
NYSE
StackOverflow
Reew
Figure 5: Example predicted time intervals, which is the expectation over Eq. 10) (upper) and RMSE (lower) on the predicting time interval to next event (section 5.2). Standard deviation is calculated among 5 train-tess, test- validation splits. The ReTweet dataset has only split so standard deviation is reported. N-SM-MPP did report RMSE retwect dataset.
as in Eq. 12). The loss function is the bog-likelihood of the training set as in (Du et al. 2016). As the model prediction, the expected duration is computed numerically from the estimated conditional distribution. The evaluation metric is the MSE of the expected duration and the actual duration to the next event. Configurations of all the models are the same in previous experiments except that: the one-hot encoded event types, {x(t;)}, }. are first passed through an embedding layer, which is a x 1 convolutional layer with channel dimension of and the resulting embedded vectors are then fed into the networks.
This task is a causal task, where current output should not depend on future input. Therefore, the CCNN configuration is adapted to the causal setting, as discussed in section 4.3
Baselines We benchmark with two baselines specialized for this type of tasks, Recurrent Marked Temporal Point Process (RMTPP) (Du et al. (016) and N-SM-MPP (Mei & Eisner 2017). N-SM- MPP is the current -fate-of-the-ate art deep learning method. For NYSE, StackOverflow and MIMIC, we directly compare to the results (Mei & Eisner (017) reported, and re-implemented RMTPP to benchmark ReTwee Configurations for CCNNs are provided in Appendix CC.4
Results and Analysis Fig. [ shows the estimated event interval (expectation of Eq. (0)) and RMSEs. The upper plots show that the predicted interval aligns well with ground truth and result in smaller RMSE in the MIMIC and Retweet dataset. In NYSE and StackOverflow, though the groud truth shows extremely fluctuation on event intervals and CCNNs fail to predict accurately as in MIMIC and ReTweet, the predicted interval still tends capture the increase and decrease trend. The lower plots compare the RMSE with the baselines. CCNN algorithms outperformed two baselines in all datasets. There is slight advantage of CCNN-th over CCNN, which verifies the effectiveness of two-hot encoding.
5.3 ADDITIONAL EXPERIMENTS
Two additional experiments on real-world data, which are predictions on Data Market and interpola- tion on speech, are presented in Appendix
6 CONCLUSION
In this paper, we have introduced CCNN for nonuniform time series with two takeaways. First, inter- polation before continuous convolution is shown to be a reasonable way for nonuniform time series. Second, learning task-specific kernels in data-driven way significantly improves the performance. There are two promising directions. First, we have focused on ID convolution, but this framework can be generalized to multi-dimensional nonuniform data. Second, while the computational complexity similar for CCNN and CNN, the runtime of the former is much longer, because of the lack of parallelization. Fast implementation of CCNN is thus another research direction.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Andrew Adams, Jongmin Back, and Myers Abraham Davis. Fast high-dimensional filtering using the permuto- hedral lattice. In Computer Graphics Forum, volume 29, pp. 753-762. Wiley Online Library, 2010.
Przemyslaw Bogacki and Lawrence F Shampine. A (2) pair of runge-uuutee kutta formulas. Applied Mathematics Letters, 2(4): 325, 1989.
Gert Cauwenberghs. An analog VLSI recurrent neural network learning continuous-time trajectory. IEEE Transactions on Newral Networks, 7(2):346-361, 1996.
Shiyu Chang, Yang Zhang, Jiliang Tang, Dawei Yin, Chang, Mark A Hasegawa-Johnsnn and Thomas Huang. Positive-umlabeled learning in streaming networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 755-764, ACM, 2016.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Baakegaw--Jhhssn and Thomas S Huang. Dilated recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 76 86, 2017.
Nan Du, Yichen Wang, Niao He, Jimeng Sun, and Le Song. Time-sensitive recommendation from recurrent user activities. In Advances in Neural Information Processing Systems, pp. 3492-3500, 2015.
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Godez-Rodriguez, and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1555-1564. ACM, 2016.
Chris Eliasmith and Charles Anderson, Neural Engineering: Computation, Representation, and Dynamics in
Neurobiological Systems. MIT press, 2004.
Ken-ichi Funahashi and Yuichi Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks, 6(6):801-800 1993.
Ramazan Gençay, Michel Dacorogna, UIrich A Muller, Olivier Pictet, and Richard Olsen. An Introduction to High-frequency Finance. Academic press, 2001.
Lcon Glass and Michael C Mackey, Pathological conditions resulting from instabilities in physiological control systems. Annals of the New York Academy of Sciences, 316(1):214-23)) 1979.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Nenworks, 4(2):251-257) 1991.
Alistair EW Johnson. Tom J Pollard, Lu Shen, I Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Authony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3:160035, 2016.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 1646- 1654, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. A clockwork RNN. In International Conference on Machine Learning, pp. 1863-1871, 2014.
Kehuang Li and Chin-Hui Lee. A deep neural network approach to speech bandwidth expansion. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pp. 4395-4399, IEEE, 2015.
Zitao Liu and Milos Hauskrecht, Learning adaptive forecasting models from irregularly sampled multivariate clinical data. In AAAI, pp. 1273-1279, 2016.
Edward N Lorenz. Deterministic nonperiodic flow. Journal of the Atmospheric Sciences, 200 -11-1110011111111111 1963.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Newral networks, 109):165-1171, 1997.
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. In Advances in Neural Information Processing Systems, pp. 6754-6764, 2017.

---

Under review as a conference paper at ICLR 2020
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased LSTM: Accelerating recurrent network training for long event-based sequences. In Advances Neural Information Processing Systems, pp. 3882-3890, 2016.
Emanuel Parzen. On estimation of probability density function and mode. The Annals of Mathematical Statistics, 33(3):1065-100 1962.
Barak A Pearlmutter: Learning state space trajectories in recurrent neural networks. Learning, 1(2), 2008.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR). IEEE, 1(2)4, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet4-4: Deep hierarchical feature learning on point sets in metric space. Advances Neural Information Processing Systems, pp. 5099 5108, 2017b.
Yiru Shen, Chen Feng, Yaoqing Yang. and Dong Tian. Neighbors do help: Deeply exploiting local structures of point clouds. arXiv preprint arXiv: 2017.
Jonathan Tapson and André van Schaik. Learning the pseudoinverse solution to network weights. Neural Networks, 45:94-100, 2013.
Jonathan Tapson, Greg Kevin Cohen, Saced Afshar, Klaus M Stiefel, Yossi Buskila, Julia Hamilton, and André van Schaik. Synthesis of neural networks for spatio-temporte spike pattern recognition and processing. Frontiers in Neuroscience, 7:153, 2013.
Yichen Wang, Nan Du, Rakshit Trivedi, and Le Song. Coevolutionary latent feature processes for contiquous- user-item interactions. In Advances in Neural Information Processing Systems, pp. 4547 4555,, 2016.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning point clouds. arXiv preprint 18X1::8800007829 2018.
Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. In AAAI, pp. 1655 1661, 2017.
Qingyuan Zhao, Murat Erdogdu, Hera Y He, Anand Rajaraman, and Jure Leskovec. Seismic: A self-exciting point process model for predicting tweet popularity. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1513-1522. ACM, 2015.
Yu Zhu, Hao Li, Yikang Liao, Beidou Wang. Ziyu Guan, Haifeng Liu, and Deng Cai. What to do next: Modeling user behaviors by TSMe-LSTM. In Twenty-Sixth International Joint Conference Artificial Intelligence (IJCAI)) pp. 3602-3608, 2017.
10

---

Under review as a conference paper at ICLR 2020
A INTERPOLATION KERNELS EXAMPLES
Piecewise Constant Interpolation:
[( t2 Jm. Xm) 1 <t -4 1441 t,]
(14)
where 1[] denotes the indicator function.
Linear Interpolation:
if0 < t- 11-1 4 1--1
[(t - t:: Xin) =
h41-1
if 4,41 t li+1
(15)
411-6
otherwise.
Sinc Interpolation:
I(t t,; Tm, (n.) aiin T( m((t-t)) /(t t.).
(16)
a
These examples are illustrated in Fig 6 Notice that in Eqs. (14 and (1). the interpolation kernels depend not only on t d-tt, t,, but also on adjacent input times, t.+1 and/or fi-1. and hence all the input times In are put in the argument. In some nonlinear interpolations, the interpolation kernel is also affected by the input values Xin-
B REPRESENTATION POWER ANALYSIS
In this section, we study how the proposed CCNN layer relates to and compares against two CNN baselines in terms of representation power. The first baseline is simply a regular convolutional layer, and the second bascline is a convolutional layer with input time intervals, At, appended to the input features, which we will call CNNT throughout.
B. CASE 1: OUTPUT TIMESTAMPS SAME AS INPUT
This subsection intuitively explains why CCNN has superior representation power to CNNT. Suppose the output timestamps are the same as input timestamps, i.e. t E Tn. Then, combining Eq. (3) with Eqs. (4) we have
v(t,) Σ (tt.) - t4; Xin)
(17)
+ 3(0; (ty ty a x(tz=*))s=000)) h=000a))
In contrast, for a CNNT layer, if we separate the convolution on time interval from the rest
y(t,)) Σ K(t)) Ctty-t--ide -
(18)
The second term of Eq. 17 represents feed-forward network on {t, ty 2(tj+x))==000 whereas the second term Eq. (88) can be regarded as a one-layer feed-forward network on
(a) Piecewise Constant
(b) Linear
(c) Sinc
Figure 6: Illustration of interpolation kemels. The red crosses denote the input data samples, The black line shows the interpolation kemel for z(t,); the gray lines show the kernels for the other points. The blue line shows the interpolated result.
11

---

Under review as a conference paper at ICLR 2020
ft, - which is equivalent to one-layer feedforward network on {t, - t1 In other words, appending the time interval feature to the convolution layer is only weak version of the CCNN bias network.
Yet a more fundamental disadvantage of CNNT lies in the first term, where the convolution kernel of CNNT, j--i. does not depend the actual time difference, but the order in which the sample arrives. This makes CNNT very vulnerable to the sampling variations. CCNN, on the other hand, has more robust kernel function for nonuniform data.
B.2 CASE 2: UNIFORM TIMESTAMPS
Both CNNT and CCNN have the same representation power as CNN under uniform sampling rate, and thus both are strict generalizations to CNN. For CNNT this trivial because the second term of Eq. (88 reduces to constant. For CCNN, we have the following theorem.
Theorem 1. Let I be the set of all functions that be represented by CCNN layer, and G be the set of all functions that can be represented by 1 x convolutional layer. Then 3 G. if the following conditions hold:
the input and output timestamps are uniform and the same, i.e. At, = At, Vi.
2. The two-hot encoding interval boundaries at multiples of At, i.e. ThK kdt.
The dimension of the two-hot vector of CCNN is no smaller than the CNN kernel length, i.e.
D Z W.
If the kernel network has hidden layers, the hidden node dimension is no smaller than W.
5. CCNN and CNN have the same receptive field size, i.e. 2L = WAt.
The kernel and bias networks do not depend on Xin-
We will only consider the 1D) case. The generalization to multi-dimensional cases is straightforward. The regular 1 x CNN performs the following operation to generate the output sequence
Proof to Theorem 7
R+(W 1)/2 Σ
M(Th)
xtt))x-i+b
(19)
i=k 1)/2
where { K (W--1)//2"" Kww-1/2 and are trainable parameters. Here we implicitly assume that W is odd. We will prove the theorem by primarily utilizing the correspondence between Equations 117 and [9).
of C G::
VK((t, h represented by the kernel network, and VB((t, -(44dh=1.ow)) repre- sented by the bias network, and thereby Vʃ E F.
By Eq. 4 and Cond. the arguments the kernel function K(-), namely the input to the kernel network, can only be set of consecutive multiples of At, i.e.
K( K([(w k) (o)
(20)
where w is an integer.
Moreover, from Eq. and Cond. 5, K(-) is non-zero iff lies in the interval [-(W 1)/2, 1)/2].
Similarly, from Eq. and Cond. the arguments the bias function 8(-) can only take one of values:
B( 3((::0)==21aa
(21)
12

---

Under review as a conference paper at ICLR 2020
Then, Eq. 19 can be made equivalent to Eq. (7 by setting
Kw (( b = ((kkt)
(22)
which means f G. Here concludes the proof that any CCNN layer can be matched by CNN layer.
G C J
Here we would only prove the case where both the weight network and bias network of CCNN has only one layer, which is the most difficult case. If either network has more than one layers, the proof easily generalizable by setting the bottom layers to identity, which is feasible because of Cond. Also, we only consider the case where the kernel network order and the bias network order are both ones, i.e.
OK 1,0 =
(23)
The proof can be generalized to larger orders by setting the additional weights to zero. In the special case defined above, the kernel network in Eq. (20) further simplified K (wwAt). The bias network in Eq. 21 is further simplified to (()).
Further, notice that the wāt, as the input to the kernel network, has to go through two-hot encoding. By Cond. 2 and 3, the two-hot encoding of wAt is one-hot vector, where only the w-th dimension activated and the rest is zero. Since the kernel network a one-layer feedforward network, denote the weight connected to the w-th dimension of the two-hot vector as Pw, then we have
(ww = P.e
(24)
V(Cw) } and b that define CNN layer, and thereby Vg E G, let
Pw
=
K
if W-1 S w S 2
(25)
0
otherwise
Cond. ensures that there are enough number of nontrivial P.s to cover all the w the case specified in line I of the equation above; and let
(0)== =
(26)
which means the bias network learns constant. Then the CCNN layer can be made equivalent to the CNN layer, i.e. g E Here concludes the proof that any CNN layer can be matched by CCNN
layer.
Thm. D iimpies that in the uniform case where the increased model complexity of CCNN is not needed, it will not harm the performance either. Replacing CNN or CNNT with CCNN will be worse off regardless of how the data are distributed in time.
As final remark, readers may argue that the improved representation power of CCNN is merely trivial result of increased model complexity, not because CCNN handles the time information more correctly. However, as will be shown the next section, even with matching model complexity, CNN and CNNT are still unable to match the performance of CCNN. CCNN does not just increase the model complexity but increases the model complexity the right way.
C EXPERIMENT SETTINGS
C.I PREDICT FUTURE SIGNAL VALUE TASK: DATA GENERATION
Three standard time series functions introduced stt).
Sine:
2nt T
(11) = sin
(27)
where = 5. The sampling time intervals follow the Poisson distribution with mean parameter
1.
13

---

Under review as a conference paper at ICLR 2020
Table Mean squared enror of prediction realworld data.
Alg.
DMI
DM2
DM3
DM4
DM5
DM6
DMM7
DM8
DMM9
DM10
DMI1
DM12
DM13
CNNT-t
0.86
0.81
0.91
0.99
0.42
0.76
0.52
0.55
0.43
0.69
0.84
0.99
0.47
ICNN-L
0.52
0.27
0.70
1.03
0.06
0.30
0.02
0.24
0.26
0.44
0.51
0.84
0.13
ICNN-Q
0.30
ICNN-C
0.63
0.29
0.71
1.02
0.06
0.30
0.04
0.25
0.29
0.43
0.56
0.82
0.13
ICNN-P
0.71
0.40
0.74
0.99
0.09
0.37
0.02
0.28
0.33
0.45
0.60
0.94
0.17
IONN-S
0.26
1.00
0.28
0.26
0.49
0.87
RNNT-th
0.79
0.27
0.70
1.06
0.05
0.38
0.42
0.20
0.33
0.48
0.57
1.01
0.11
CCNN-t
0.49
0.23
0.65
1.00
0.05
0.27
0.04
0.22
0.29
0.41
0.50
0.80
0.12
Mackey-Glac (MG): a chaotic time series inspired by biological control systems (Glass & Mackey 1979). MG is the solution to the following delay differential equation.
x(t - 73r(t)
=(1) = 1+ x(t T)"
(28)
where å denotes dx dl. We choose 0.2, 17, 10, and 0.2. There is no closed-form
solution to Eq. (28). so the Kunge-KKuta method Bogacki & Shampinc (9899) is applied to obtain a discrete numerical solution, which is a uniform sequence with sampling interval At. A set of N-sample short uniform sequences generated from the long sequence by sliding window with window shift of one. The uniform sequences are subsampled into nonuniform sequences by choosing M (M N) sample points uniformly random I | 2, N 42, M 14 this dataset. The timestamps are normalized such that the expected interval of the nonuniform sequences is one.
Lorenz: simplified system describing the 2D) flow of fluid (Lorenz 1963). The Lorenz function is the solution to the following differential equation system.
+(() (v(1) - sx())
i(t) =(())(())+aat)))
(29)
(t) = x(t)v(t) - 6z(()
We choose r 28, b 8/3. The nonuniform subsequences generated the same way as in MG with At = 0.05. Only x(f) is observed and predicted.
C.2 PREDICT FUTURE SIGNAL VALUE TASK HYPERPARAMETERS
As mentioned, the hyperparameters are set such that all the architectures share the same number of layers, receptive field size, and number of parameters, All the hidden activation functions ReLU, and all the output activation functions are linear.
CNN: CNN has two x 7 layers, and the number of hidden channels is 84. The total number of parameters is 1,261.
CNNT: CNNT has two 7 layers, and the number of hidden channels is 60. The total number of parameters is 1,261. The sampling time intervals are appended as input features. The time interval appended to the last sample of the sequence is the difference between the prediction time and the time of the last sample.
The two-hot encoding interval width 8 0.5, and the two-hot vector dimension is 14. The network has two 1 layers, and the number of hidden channels is 10. The total number of parameters 1,261.
CNNT-th:
ICNN takes the interpolated signal at f.++1 k k = 1.... 13 as input. The network has two 1 x 7 layers, and the number of hidden channels 84. The total number of parameters is 1,261.
ICNN:
RNNT:RNNT and RNNT-th have two layers and the number of hidden channels is 32. The total number parameters 1153 for RNNT and 1633 for RNNT-th.
CCNN: CCNN has two layers. The first layer is CCNN layer with output timestamps at 4,+1 k ...,.....1.
= 1. 13. The bias network has two layers and 72 hidden channels. Its order Og is 7. The
Equiivaent Poisson process conditional the event that the number of accurrences by the time N is M.
14

---

Under review as a conference paper at ICLR 2020
kemel network has two layers and hidden nodes. Its order OK is 3. The kernel length L = 3. The second layer is a regular x 7 convolutional layer. The total number of parameters is 1,273.
CCNN-th: CCNN-th has almost the same structure as CCNN, except that the number of hidden channels 36. The total number of parameters is 1,261.
C.3 DESCRIPTION OF REAL-WORLD DATASET FOR PREDICTING INTERVALS TO NEXT EVENT
Stackoverflow:
The dataset contains 26535 training samples, 3315 test samples, and 3315 validation samples. Each sequence represents the history awarding badges to a single user. The sampling timestamps are the time of awarding each badge, and the signal value is the type of the badge (e.g.. Guru, Nice Question, Great Answer, etc.). There are 22 types of badges total.
The dataset contains 20000 training samples, 1000 test samples, and 1000 validation samples. The sequence is the history of tweet being re-posted by other users. According to the number of followers of the users retweeted the post, the label of each retweet, in the whole retweet history, is one of the 3 the user groups.
Retweet:
The dataset is the collection of clinical visit history. The training set contains 2925 se- quences; the test and validation set contains 312 each. The diagnosis are filtered preserve only top-10 common diseases the label.
MIMIC:
NYSE The dataset is book order data collected from NYSE of high- tigh-frreuency transaction in one day. The dataset contains 2 types of events (sell and buy) with 298710 training sequences, 33 $3190 testing sequences, and 82900 validation sequences.
C.4 CONFIGURATION OF PREDICTING TIME INTERVAL TO NEXT EVENT
The input sequence length to NYSE, Stackoverflow, and ReTweet dataset is 13, and the CCNN uses two 7 kernels with 16 filters for each. MIMIC contains only very short sequences, so CCNN uses
two 1 x 2 kernels and predicts only based on past 3 events.
D ADDITIONAL EXPERIMENTS
D.1 PREDICT FUTURE SIGNAL VALUE ON REAL-WORLD DATASET WITH MISSING DATA
In order to test the advantage CCNN in real-world scenarios with missing observations, 13 time- series datasets are randomly chosen from the Data Marke named DM1 through DM13. A brief description of these datasets given below. Each dataset consists of a univariate uniform time series, which is split into training, test and validation sequences by ratio of 6:2:2. Nonuniform subsequences are generated the same way as in MG with N = 28 and M = 14. All of the data monthly data. The network configurations are the same those in the simulated experiment. In particular, the receptive field size is set to seven, which means the prediction is based on an average of 14 months of historic data. This should lend adequate information for prediction.
DM1: Australia monthly production of cars and station wagons from Jul 1961 to Aug 1995 The
total length of the sequence is 414.
sequence
DM2: Monthly data on Clearwater River at Kamiah, Idaho from 1911 to 1965 9 The total length of the sequence is 604.
DM3: Monthly data on James River at Buchanan, VA from 1911 to 1960 5 The total length of the sequence is 604.
https:/ddttamarket.com
hwtp:/aww.....iaaaiii.i.tiiittta 19955 1ls==22f&ddisplayylli
line
15

---

Under review as a conference paper at ICLR 2020
DM4: Mean monthly precipitation from 1907 1972 The total length of the sequence is 796.
DM5: Mean monthly temperature from 1907 to 1972 The total length of the sequence is 796.
DM6: Monthly data Middle Boulder Creek at Nederland, CO from 1912 1960 8 The total length of the sequence is 592.
DM7: Monthly electricity production in Australia (million kilowatt hours) from Jan 1956 Aug 1995 The total length of the sequence 480.
DM8: Monthly flows of Chang Jiang (Yangtze River) Han Kou, China from 1865 to 1979 The total length of the sequence is 1372.
DM9: Monthly production of clay bricks (million units) from Jan 1956 to Aug 1995 1T The total length of the sequence is 480.
DM10: Monthly rain in Coppermine (mm) from 1933 to 1976 The total length of the sequence is 532.
DM11: Monthly riverflow (cms) in Pigeon River near Middle Falls, Ontario from 1924 1977 The total length of the sequence 640.
DM12: Monthly riverflow (ems) in Turtle River Mine Centre, Ontario from 1921 to 1977 The total length of the sequence is 676.
DM13: Monthly temperature in England (F) from 1723 to 1970 The total length of the sequence is 2980.
Table 2 shows the mean squared prediction errors. CCNN-th maintains its lead most datasets. Where does not, different ICNNs alternately take the lead by small margins. Here are two comments. First, note that the kernel length in ICNN-C and ICNN-Q is much larger than Lx. it falls beyond the representation power of CCNN. Nevertheless, this result shows that an interpolation kernel within the scope of Eq. ①I usually suffices outperform the standard interpolation methods that fall beyond. Second, unlike in the simulated test, ICNN-L generally performs better than ICNN-C, which emphasizes the importance of choosing suitable interpolation scheme for each task. CCNN, with its ability to choose its own kernels, avoids such trouble.
D.2 SPEECH INTERPOLATION
Since CCNN is motivated by interpolation, it is insightful to see CCNN's performance in interpolation tasks. The speech interpolation task involves restoring the high-resolution speech signal from the downsampled signal.
THE DATASET
To mitigate the complexity in directly working on speech waveforms, the sampling rate of the high- kHz, and that of the downsampled signals kHz. Three different downsampling schemes are tested. The first scheme, called smiform filtered, uniformly downsamples the speech kHz after passing it to an anti-aliasing filter. The second scheme, called uniform unfiltered, uniformly downsamples the signal without the anti-aliasing filter. The third scheme, called nonuniform, randomly preserves half of the speech samples, and thus the resulting signal has an average sampling rate of 2 kHz.
resolution speech
http::/ddtaaartttt..mmmmaa/s///s -onthly-tepperruuure 1907- 1972
monthly propillyyppreppiittiioioooo 1907- 1972
httpp///daamnnkkkccmm////tt//t// bowddedeeeedddddde-eeeeeeee aeweenteenaddddeeee 1960
htt.s/awwsaaaaaiilooooootttott-a www.mlloooooooottooouoosseeo kilowatt-hours-jans 195--ja---1---1111111
1aw-an--95-uuu--- 1016-uug-1997 1995
190--ott---9--999--- 1924-1977
http://daiikkoooooo///////////ia -et---ooo------111-11199---- 1921-1977
16

---

Under review as a conference paper at ICLR 2020
Table 3: Signal-to-Noie Ratio (dB) in Speech Interpolation.
Uniform
Alg.
Non-uniform
filtered
non-filtered
Speech DNN
9.13
Speech DNN (CP)
13.64
ICNN-C
9.74
7.49
2.33
ICNN-Q
9.66
5.67
2.77
ICNN-L
9.72
5.81
3.16
ICNN-P
3.17
2.63
1.82
ICNN-S
9.62
5.90
2.83
CCNN-th
9.61
7.80
6.58
Speech DNN does not work with non-filtered down-sampled signals and nonuniformly down-sampled signals.
Our dataset consists of one hour of lab-recorded speech one speaker reading structured composite sentences. We use 80% of the dataset as training. 10% validation, and the rest of 10% as test. The high-resolution speech is chunked into 40-point sequences without overlap, and the corresponding downsampled speech into 20-point sequences.
CONFIGURATIONS
Similar to the prediction experiment, the ICNN approaches interpolates the low-resolution speech into high-resolution speech (4 kHz) before it is fed to a two-layer regular CNN. A similar practice has also been adopted in Kim et al. (2016). CCNN also has two layers. The first layer outputs at the uniform timestamps at the rate of kHz, and the second layer is a regular convolutional layer. Detailed configurations shown below. Again, the hyperparameters have the same number of layers, receptive field size and model complexity. The activation functions for CCNN and ICNN are both sigmoid. The hyperparameters are detailed below.
set such that the two architectures
The timestamps are normalized such that the sampling interval of the original 4 kHz speech is 0.5. CCNN-th has two layers. The first layer is CCNN layer that outputs the timestamps uniformly distributed at the rate of 4 kHz. The number of hidden channels is 16. The bias network has one layer, and the order OB 28. The bias kernel network has one layer, and the order OK The kernel length Lx= = 10. The two-hot encoding time interval 5 = 0.5. The second layer is a x 28 convolutional layer.
CCNN-th:
ICNN-C: The CNN contains two 1 x 28 convolutional layers with 80 number of hidden filters.
Speech DNN & Speech DNN CP: Speech DNN converts both the high resolution and downsampled speech into amplitude spectra using FFT with 64ms window length and 48ms window shift. Speech DNN has 3 hidden layers, each of which has 1024 hidden nodes. Because our temporal resolution is half of that in Li & Lee (0155), our hidden node number is half of that in Li & Lee (2015) too. The number of parameters Speech DNN is around 2.3 x 10 which much larger than that CCNN and the ICNN baseline models (both have around 5 10 parameters).
Since sample- based speech interpolation methods have yet to achieve the state-of-the-art,, we also include a spectrum-based -ased benchmark from Li & Lee (2015) called Speech DNN. Speech DNN only works on the uniform filtered case. It predicts the higher half of the amplitude spectrum of the reghurtsol speech of that of the low-resolution speech. The flipped phase spectrum of the down- sampled speech is used as an estimate of the higher half of the phase spectrum of the high-resolution speech.
Since the phase spectrum estimate of Speech DNN can be inaccurate, we introduced another version, called Speech DNN with cheated phase (CP), where the ground truth phase spectrum is applied. Note that this version is given too much oracle information to be fairly compared with. Nevertheless, we will include its results for reference.
RESULTS AND ANALYSTS.
Tab. shows the Signal-to-Non Noise Ratio (SNR) of the signals recovered from different input signals by different models. The Speech DNN with cheated phase yields the best SNR, because it uses the phase information from ground truth. However, the Speech DNN without cheated phase has similar
17

---

Under review as a conference paper at ICLR 2020
Sendl Magntuse
0.1
Signel Magntadk
10.1
Groundruth
0.1
Inputt
10
20
50
60
70
10
20
30
60
70
Sample Index
Sample Index
(a) By CCNN
(b) By ICNN
Figure 7: Examples for CCNN and ICNN in restoring nonuniformly down-sampled speech. CCNN generates better approximation especially crests and troughs.
performance to CNN and CCNN, even though it has much more weights, largely because of the inaccurate phase estimates.
As for the comparison between CCNN and ICNN, they have similar SNR under uniform sampling cases. This verifies that both architectures have similar representation power given uniform data. However, CCNN has much higher SNR than ICNN in nonuniform case. One important reason is that CCNN, by construction, is aware of whether neighboring samples are dense or sparse, and outputs robust interpolation kernels accordingly, despite the variation in sampling patterns; whereas CNN is unable to deal with various random sampling instances.
Fig. 7 shows example of the restored signal from the nonuniform samples by CCNN and CNN respectively. CCNN can restore some spikes (e.g. ones at 25 and 40) even without an input sample point in the spike, because CCNN can learn the continuous kernels and restore the original spikes. CNN model does poorly in restoring spikes even when there are input sample points in the spikes.
18


Reviews:
[{"review_id": "r1lPKXh6KB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A method that was proposed by authors deals with a problem of non-uniform data in time series. One of ways to deal with this problem is interpolate input signal between data points. In signal processing a standard way to interpolate is to apply a convolution with a kernel. This operation, by itself, is a non-trivial, since we lack any information about signal's spectrum (and do not know optimal kernel). Thus, the authors propose to search for a kernel in a form neural network. To this term they also add bias term which is also a neural network. \n\nBasic idea of the paper seems promising, but reported results are only partial. Since a paper is experimental, i.e. no theory at all, then the main judgement should be based on experimental results. They are not convincing, as CCNN is compared with methods that can not be called state-of-the-art.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SylsLane9H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1.\tThe motivation of continuous convolution is not very clear, can the authors please motivate? To my understanding this is just to handle inputs with unequal time steps, but that can be handled multiple ways, why not just naively resample?\n2.\tThe proposed network was defined as continuous convolution followed by the standard convolution. Why not just stack multiple continuous convolutions?\n3.\tContinuous convolution should be a general case for standard convolution, can authors explicitly show it?\n4.\tAnother way to handle unequal timesteps is by using dilated convolution, can authors please comment how they differ, pros and cons etc.?\n5.\tTwo hot encoding seems another way to discretize, no?\n6.\tThe experiments section is rather weak, CCNN seems to have a lot of spikes in prediction, e.g., in Fig. 5.\n7.\tIt’s very strange why two hot encoding does not perform that well, while reading the method section, it seems very obvious to take two ends of an interval, in that way two hot encoding seems logical.\n\nOverall it seems like an easy extension with a lot of parts not well-justified. Also I don't clearly have a well-grounded motivation for a continuous convolution.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BkeIvRLRYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a continuous CNN model to accommodate the nonuniform time series data. The model learns the interpolation and the convolution kernel functions in an end-to-end manner, so that it can capture the signal patterns and be flexible. A layer has three networks, which learn a kernel function to represent the combination of interpolation and convolution, a bias function to represent the error correction with convolution, and then produce the output based on them. The authors introduce two assumptions and a two-hot encoding scheme for the input to control the model complexity. The paper also introduces an application of the proposed CCNN by combing with temporal point process. Experiments on simulated data compares the proposed method with some degenerative baselines show the advantage of learning the interpolation and the two-hot encoding configuration. The authors compare the performance on time interval prediction task based on real world dataset to show the model produces a better history embedding for the task.\nOverall, the paper has some incremental improvements on the existing methods that dealing with the nonuniform time series data. Instead of using preset interpolation kernels, the proposed model can learn it with the convolution in a data-driven manner.\nThe paper includes clear explanation on module structure and detailed experiment settings.\nThe experiments of signal value prediction support the claims of the advantages of the proposed model.\nThe notation in the caption of figure 1 is a little confusing: is x(t_i’) the same as hat x(t) in the algorithm?\nIt is good that the related works section mentioned the adapted RNNs that are used as baselines in the real-world dataset experiment, and the differences between the proposed model and the related SNNs are introduced.\nHowever, this section and the introduction can be better organized to distinguish the novelty and the contribution of the work.\nIn page 7, the purpose of the reference in the sentence “The time information is either two-hot encoded (Adams et al., 2010) (CCNN-th), or not encoded (CCNN).” is not very clear.\nIt will be better if there are a little more analysis of the experiment on predicting time intervals to next event.\nThe upper plots in the figure may be not convincing enough to support the claims.\nThe advantage of two-hot encoding seems subtle in the figure 5. Is there any reason for the significantly higher deviation of CCNN-th in StackOverflow?\nSometimes the usage of “CCNN” is not clear, for example, the experiment on speech interpolation compares the “CCNN-th” method with baselines, but uses “CCNN” in the analysis. Also, it could be better to show the “CCNN-th” result in the upper plots of figure 5 instead of “CCNN”.\nMinor comment:\nThere are some typos in the paper, for example, missing the right parenthesis in page3 “(refer to Appendix A.1”, in page4 section 4.1 “According to Eq.(4), the input is …”.\n“The left plot shows” in the last line of the caption of figure 3 should be “right”.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a continuous CNN model that can handle nonuniform time series data. It learns the interpolation kernel and convolutional architectures in an end-to-end manner, which is shown to achieve higher performance compared to naïve baselines. 
All reviewers scored Weak Reject and there was no strong opinion to support the paper during discussion. Although I felt some of the reviewers’ comments are missing the points, I generally agree that the novelty of the method is rather straightforward and incremental, and that the experimental evaluation is not convincing enough. Particularly, comparison with more recent state-of-the-art point process methods should be included. For example, [1-3] claim better performance than RMTPP. Considering that the contribution of the paper is more on empirical side and CCNN is not only the solution for handing nonuniform time series data, I think this point should be properly addressed and discussed. Based on these reasons, I’d like to recommend rejection. 

[1] Xiao et al., Modeling the Intensity Function of Point Process via Recurrent Neural Networkss, AAAI 2017.
[2] Li et al., Learning Temporal Point Processes via Reinforcement Learning, NIPS 2018.
[3] Turkmen et al, FastPoint: Scalable Deep Point Processes, ECML-PKDD 2019.

Decision: Reject