A Temporal Kernel Approach for Deep Learning with Continuous-time Information
Authors: Da Xu
Chuanwei Ruan
Evren Korpeoglu
Sushant Kumar
Kannan Achan
Venue: ICLR 2021 Poster (2021)
Keywords: Kernel Learning
Continuous-time System
Spectral Distribution
Random Feature
Reparameterization
Learning Theory

Abstract: Sequential deep learning models such as RNN, causal CNN and attention mechanism do not readily consume continuous-time information. Discretizing the temporal data, as we show, causes inconsistency even for simple continuous-time processes. Current approaches often handle time in a heuristic manner to be consistent with the existing deep learning architectures and implementations. In this paper, we provide a principled way to characterize continuous-time systems using deep learning tools. Notably, the proposed approach applies to all the major deep learning architectures and requires little modifications to the implementation. The critical insight is to represent the continuous-time system by composing neural networks with a temporal kernel, where we gain our intuition from the recent advancements in understanding deep learning with Gaussian process and neural tangent kernel. To represent the temporal kernel, we introduce the random feature approach and convert the kernel learning problem to spectral density estimation under reparameterization. We further prove the convergence and consistency results even when the temporal kernel is non-stationary, and the spectral density is misspecified. The simulations and real-data experiments demonstrate the empirical effectiveness of our temporal kernel approach in a broad range of settings.

Published as a conference paper at ICLR 2021
A TEMPORAL KERNEL APPROACH FOR DEEP LEARN- ING WITH CONTINUOUS-TIME TIME INFORMATION
Da Xu
Walmart Labs
Sunnyvale, CA 94086, USA DaXu51 800 gma i com
Chuanwei Ruan Instacart
San Francisco, CA 94107, USA
Ruanchuanwei gma com
Evren Korpeoglu, Sushant Kuamr, Kannan Achan
Walmart Labs
Sunnyvale, CA 94086, USA
[EKorpeoglu, SKumar4, KAchan] @walmartlabs.com
ABSTRACT
Sequential deep learning models such as RNN, causal CNN and attention mech- anism do not readily consume continuous-time information. Discretizing the temporal data, as we show, causes inconsistency even for simple continuous processes. Current approaches often handle time in heuristic manner to be con- sistent with the existing deep learning architectures and implementations. In this paper, provide principled way to characterize continuous-tin deep learning tools. Notably, the proposed approach applies to all the major deep learning architectures and requires little modifications to the implementation. The critical insight is to represent the continuous-time system by composing neural networks with a temporal kernel, where we gain our intuition from the recent advancements in understanding deep learning with Gaussian process and neural tangent kernel. To represent the temporal kernel, we introduce the random feature approach and convert the kernel learning problem to spectral density estimation under reparameterization. We further prove the convergence and consistency re- sults even when the temporal kernel is non-stationary, and the spectral density is misspecified. The simulations and real-data experiments demonstrate the empirical effectiveness of our temporal kernel approach in broad range of settings.
systems using
INTRODUCTION
Deep learning models have achieved remarkable performances in sequence learning tasks leveraging the powerful building blocks from recurrent neural networks (RNN) (Mikolov et al. 2010). long- short term memory (LSTM) (Hochreiter & Schmidhuber 1197). causal convolution neural network (CCasalCNN/WaveNee (Oord et 2016) and attention mechanism (Bahdanau et al., 2014 Vaswani et al. 2017). Their applicability to the continuous- time data, on the other hand, is less explored due to the complication of incorporating time when the sequence is irregularly sampled (spaced). The widely-adopted workaround is to study the discretized counterpart instead, e.g. the temporal data is aggregated into bins and then treated as equally-spaced, with the hope to approximate the temporal signal using the sequence information. It is perhaps without surprise, as we show in Claim I that even for regular temporal sequence the discretization modifies the spectral structure. The gap can only be amplified for irregular data, so discretizing the temporal information will almost always
'The work done when the author was with Walmart Labs.

---

Published as a conference paper at ICLR 2021
introduce intractable noise and perturbations, which emphasizes the importance to characterize the continuous -ime information directly. Previous efforts to incorporate temporal information for deep learning include concatenating the time timespan to feature vector (Choi et al. 2016 Lipton
et al. 2016, [i et al. 2017b), learning the generative model of time series as missing data problems
(Soleimani et al. [20 Futoma et al. 2017), characterizing the representation of time (Xu et al. 2019
2020 Du et al. (2166) and using neural point process (Mei & Eisner 2017 Li et al., 20188). While they
provide different tools expand neural networks to coupe with time, the underlying continuous-time
system and process are involved explicitly or implicitly. As a consequence, it remains unknown in what way and to what extend are the continuous- learning model. Explicitly characterizing the continuous-tine system (via differential equations), on the other hand, is the major pursuit of classical signal processing methods such as smoothing and filtering (Doucet & Johansen 2009] Särkkä 2013). The lack of connections is partly due to the compatibility issues between the signal processing methods and the auto-diffcrential gradient computation framework of modern deep learning. Generally speaking, for continuous- systems, model learning and parameter estimation often rely on the more complicated differential equation solvers (Raissi & Karniadakis] 2018 Raissi et al. 2018a). Although the intersection of neural network and differential equations is gaining popularity in recent years, the combined neural differential methods often require involved modifications both the modelling part and implementation detail
signals interacting with the original deep
Chen et al. 2018 Baydin et al. 2017).
Inspired by the recent advancement in understanding neural network with Gaussian process and the neural tangent kernel (Yang] [219)] Jacot et al.) 20188), we discover a natural connection between the continuous- time system and the neural Gaussian process after composing with a temporal kernel. The significance of the temporal kemeel is that it fills in the gap between signal processing and deep learning: we can explicitly characterize the continuous- time systems while maintaining the usual deep learning architectures and optimization procedures. While the kernel composition is also known for integrating signals from various domains (Shawe-Taylor et al. 20044), we face the additional complication of characterizing and learning the unknown temporal kernel in data-adaptive fashion. Unlike the existing kernel learning methods where at least the parametric form of the kernel is given (Wilson et al. 2016), we have little context on the temporal kernel, and aggressively assuming the parametric form will risk altering the temporal structures implicitly just like discretization. Instead, we leverage the Bochner's 's theorem and its extension (Bochner) 1948 Yaglom| 1987) to first covert the kernel learning problem to the more reasonable spectral domain where we can direct characterize the spectral properties with random (Fourier) features. Representing the temporal kernel by random features is favorable as we show they preserve the existing Gaussian process and NTK properties of neural networks. This desired from the deep learning's perspective since our approach will not violate the current understandings of deep learning. Then we apply the reparametrization trick (Kingma Welling 2013), which is standard tool for generative models and Bayesian deep learning, to jointly optimize the spectral density estimator. Furthermore, we provide theoretical guarantees for kernel learning approach when the temporal kernel is non-stationary, and the spectral density estimator is misspecified. These two scenarios are essential for practical usage but have not been studied in the previous literature. Finally, we conduct simulations and experiments on real-world continuous-time sequence data to show the effecttieness of the temporal kernel approach, which significantly improves the performance of both standard neural architectures and complicated domain-specific models. We summarize our contributions as follow.
the
We study novel connection between the continuous-tin time system and neural network via the composition with a temporal kernel.
We propose an efficient kernel learning method based on random feature representation, spectral density estimation and reparameterization, and provide strong theoretical guarantees when the kernel is nonstationary and the spectral density is misspeficied.
We analyze the empirical performance of our temporal kernel approach for both the standard and domain-specifi deep learning models through real-data simulation and experiments.
2 NOTATIONS AND BACKGROUND
We use bold-font letters to denote vectors and matrices. We use x, and (x, t) interchangeably to denote time-sensitive event occurred at time with & E T [0, mm Neural networks are denoted
2

---

Published as a conference paper at ICLR 2021
by x)),)), where & i is the input where diameter(X)) Σ L, and the network parameters O are sampled i.i from the standard normal distribution at initialization. Without loss of generality, we study the standard L-layer feedforward neural network with its output the hth hidden layer given by f(h)) E Rd, We use and e(t) to denote Gaussian noise and continuous- cimim Gaussian noise process. By convention, we use & and and to represent the tensor and outer product.
2.1 UNDERSTANDING THE STANDARD NEURAL NETWORK
We follow the settings from acot et al. 2018 Yang 2019) to briefly illustrate the limiting Gaus- sian behavior of f(0, x) initialization, and its training trajectory under weak optimization. As d1, dj....,, dr 00, f (()) tend in law i..d Gaussian processes with covariance Ee E I& xda; ((h) (0,00"00 )²). which we refer to as the neural network kernel to distinguish from the other kernel notions. Also, given training dataset {x,, 3/ Ja let (())))
(S(06)) (f(0(),x))), ..)).....(........ xn)) be the network outputs at the 8hh training step and y (y1, (y1,-.., Un)- Using the squared loss for example, when training with infinitesimal learning rate, the outputs follows: of ((())) ((()))/ds
(s) x (f (((+)) y), where (s) the neural tangent kernel (NTK). The detailed formulation of Σ and (s) are provided in Appendix A.2 We introduce the two concepts here because:
1. instead of incorporating time to f(0, x), which is then subject to its specific structures, alternatively consider an universal approach which expands the A to the temporal domain such as by composing it with time-aware kernel?
2. When jointly optimize the unknown temporal kernel and the model parameters, how can we preserve the results on the training trajectory with NTK?
In our paper, we show that both goals are achieved by representing a temporal kernel via random features.
2.2 DIFFERENCE BETWEEN CONTINUOUS AND ITS DISCRETIZATION
We now discuss the gap between continuous time process and its equally-spaced discretization. We study the simple univariate continuous-tin system f(t):
f(t d + ag
d.((t)
+ a al)(() (l)
dt
A discretization with fixed interval is then given by: f1y = f( x interval) for = 1,2,.... Notice that f() is second-order auto-regressive process, so both f(t) and fly are stationary. Recall that the covariance function for stationary process is given by k(t) cov(f(to). f(to + t)), and the spectral density function (SDF) is defined s(w) = - exp(- (i)-)
Claim 1. The spectral density function for f(t) and f1S) are different.
The proof is relegated to Appendix A.2.2 The key takeaway from the example is that the spectral density function, which characterizes the signal the frequency domain, is altered implicitly even by regular discretization in this simple case. Hence, we should be cautious about the potential impact of the modelling assumption, which eventually motivates to explicitly model the spectral distribution.
METHODOLOGY
We first explain our intuition using the above example. If we take the Fourier transform on bn
and rearrange terms, it becomes: f((w) (iw)2 an (iw) + E(iw), where f(iw) and E(iw)
the Fourier transform of f(t) and e(t). Note that the spectral density of Gaussian noise process constant, i.c. (²(ww)?² = Po. so the spectral density of f(t) is given by: sø, (w) = Po (o + au (iw) + o1) where linear dynamic system defined in (I). The subscript T' is added to distinguish from the parameters of the neural network. The classical Wiener- Kimner-Khinchhe Khinchin theorem (Wiener al., 1930) states that the
O, (oo, 41, bu] to denote the parameters of the
3

---

Published as a conference paper at ICLR 2021
(a)
(b)
E (xxxxe)
kx'x,,xx',e)
Y(G-))
Y(O
V(h+))
BE-NN
BE-NN
Hedden Factors PX (X.().))
x (x,,x)
GRU Call
GRU
GRU Cell
Cell
X(G)
LgLs
Feature
(G
((
Feature X
Time (t)
Figure (a). The relations between neural network kernel (h), temporal kernel Kr and the neural-temporal kernel !h) (b). Composing single-layer RNN with temporal kernel where the hidden out from GRU cells become ((h) t) (x) Φ(x, t). We use the RF-INN blocks to denote the random feature representation parameterized by INN. The optional outputs y(t) can be obtained a way similar to f(h) (x, t).
covariance function for (tt,, which a Gaussian process since the linear differential equation is linear operation on E(f), is given by the inverse Fourier transform of the spectral density:
2T / so, (w) exp(iwt)dw,
1
(t(,,t') == kor
(t' - (t'
We defer the discussions on the inverse direction, that given kernel kg. (t' -t) we can also construct continuous-ting system, Appendix AA3.I] Consequently, there is correspondence between the parameterization of a stochastic ODE and the kernel of a Gaussian process. The mapping is not convenient way parameterize continuous-
necessarily one-to to-one; however, it may lead to time process alternatively using deep learning models, especially knowing the connections between neural network and Gaussian process (which we highlighted in Section 2.1
To connect the neural network kernel (h) (e.g. for the th layer of the FFN) to continuous-time system, the critical step to understand the interplay between the neural network kernel and the temporal kernel (e.g. the kernel in )2):
the neural network kernel characterizes the covariance structures among the hidden repre- sentation of data (transformed by the neural network) at any fixed time point;
the temporal kernel, which corresponds to some continuous-time system, tells how each static neural network kernel propagates forward in time. See Figure Ta for visual illustration.
Continuing with Example
I straightforward construct the integrated continuous-time system as:
fx, t) di2
df(x,t) (x) 6o (o())) t), e(x, t to) N (M,,, Vto E T,
(x)
ta (x)
dt
(3)
where we use the neural network kernel Σ to define the Gaussian process e(x, f) on the feature dimension, so the ODE parameters are now functions of the data well. To see that (3) generalizes the htth layer of a FFN the temporal domain, we first consider az a2(x) a1 (x) and ag(x) = (x).
Then the continuous-tim process f(x, t) exactly follows f(h) at any fixed time point t, and its trajectory on the time axis is simply a Gaussian process. When a (x))xx2(x) * 0. f(x, () still
matches (h) at the initial point, but its propagation on the time axis becomes nontrivial and is now characterized by the constructed continuous-time system. We can easily the setting to incorporate higher-order terms:
f(x,t) d
d' d'x((x, t)
On (x)
(xx))
bx.(x)
by (x(x)((Xx))
dmm
Keeping the heuristics in mind, an immediate question is what is the structure of the corresponding kernel function after combine the continuous-time system with the neural network kernel?
4

---

Published as a conference paper at ICLR 2021
Claim 2. The kernel function for f(x.1) 1) in 4 is given by: E.(o1 (x, 1; x', (1) ko, 1x,1;x111 1)) E(h (x, x)), where 0r is the undertying parameterization of { {a:(t) ) and (b(()) as functions m are scalars, >0///) (x,,:'', = (t,t)) (x,
of When {ai) and {b
We defer the proof and the discussion on the inverse direction from temporal kernel to continuous- time system to Appendix [A.3.T Claim 2 shows that it is possible to expand any layer of a standard neural network to the temporal domain, as part of a continuous-time system using kemel composition. The composition flexible and can happen at any hidden layer. In particular, given the temporal kernel K and neural network kernel E(h,
obtain the neural neural-temporal kernel on X' x T: E/h = diag (k)) & Kr), where diag(.) is the partial diagonalization operation on X::
(5) The above argument shows that instead of taking care of both the deep learning and fontinuous-timg system, which remains challenging for general architectures, we can convert the problem to finding a suitable temporal kernel. We further point out that when using neural networks, we are parameterizing the hidden representation (feature lift) in the feature space rather than the kernel function in the kernel space. Therefore, to give a consistent characterization, we should also study the feature representation of the temporal kernel and then combine with the hidden representations of the neural network.
D// (x,1;xx,1) = E (x, (x,x') (x, 1;x', (').
3.1 THE RANDOM FEATURE REPRESENTATION FOR TEMPORAL KERNEL
We start by considering the simpler case where the temporal kernel is stationary and independent of features: (r(1, (') = h:(l- - l), for some properly scaled positive even function k(-). The classical Bochner's 's theorem (Bochner 1948) states that:
s(t' t) / i(' i(t'-t) ds(w), for some probability density function s on R.
(6)
where s()) is the spectral density function we highlighted in Section 2.2 To compute the inte- gral, we may sample (wi,..., Wm) from s(w) and use the Monte Carlo method:
vb((' - () Pd
E ttw, Since H(( cos ((1' - ())) isin ((1' - f)):), for the real part, we let:
x(t) = - cos(twu). sin (owi)tw))))))))) sin(twi)..... .o........ssstp..p. ) sin sinnttea (twm)]]
(7)
Vm
and it is easy to check that (t('' tt) = (Ø()), Φ(t)). Since (t) constructed from random samples, we refer to it as the random feature representation of K7. Random feature has been extensive studied in the kernel machine literature, however, we propose a novel application for random features to
parametrize unknown kernel function. A straightforward idea is parametrize the spectral density function s(w), whose pivotal role has been highlighted in Section 2.2 and Example !]
Suppose 0r is the distribution parameters for s(w). Then de,, (t) is also (implicitly) parameterized by Or through the samples (w(Or):) 1" from s(w). The idea resembles the reparameterizzttoo trick for training variational objectives (Kingma & Welling 2013), which we formalize in the next section. For now, it remains unknown if we can also obtain the random feature representation for non- stationary kernels where Bochner's theorem is not applicable. Note that for a general temporal kernel Kr(x, xx,', t'). in practice, it is not reasonable assume stationarity specially for the feature domain. In Proposition] [] we provide a rigorous result that generalizes the random feature representation for nonstationary kernels with convergence guarantee.
Proposition 1. For any (scaled) continuous non-stationary PDS kernel Kr on X x T, there exists a joint probability measure with spectral density function s(w1, W2), such that Kr ((x, f), (x', {t))
Exiny w))) [o(x, t)T (0xx(ttt)) ,t']] where Φ(x, t) is given by:
2/m cos (1x, w).k) + cos ([x, w2.)) sin ([x, ww1,.) sin ([x, ([T ...],
(8)
Here, (w..) are the m samples from (2)).))) When the sample size m N
8(d+1) log ((d) (1222 m (2/e 0//c)
2042 w4s
15). with probability at least 1 - for any > 0.
sup
Kr ((x, t), (x,,))
+2) Se,
(9)
(x)
5

---

Published as a conference paper at ICLR 2021
where og is the second moment of the spectral density function s(w1, w2) and C(d) is constant.
We defer the proof to Appendix A.3.2 It is obvious that the new random feature representation in is a generalization of the stationary setting. There are two advantages for using the random feature representation:
the composition in the kernel space suggested by 5 is equivalent to the computationally efficient operation f()) (x) o Φxx, t) in the feature space SShaw--Tallr et al. 2004):
we preserve a similar Gaussian process behavior and the neural tangent kernel results that we discussed in Section 2.1 and we defer the discussion and proof to Appendix A.3.3
In the forward- passing computations, we simply replace the original hidden representation (() (x) by the time-aware representation f (x))(x) ((xx x t). Also, the existing methods and results on analyzing neural networks though Gaussian process and NTK, though not emphasized in this paper, can be directly carried out to the temporal setting as well (see Appendix (A.333 AA..3)..
3.2 REPARAMETERIZATION WITH THE SPECTRAL DENSITY FUNCTION
We now present the gradient computation for the parameters of the spectral distribution using only their samples. We start from the well-studied case where s(w) is given by a normal distribution N Σ) with parameters Or [u, Σj. When computing the gradients of Ôr, instead of sampling from the intractable distribution sww), reparameterize each sample w; via: E + μ4, where is sampled from a standard multivariate normal distribution. The gradient computations that relied on is now replaced using the easy-to- so-yppl and Or [M. Σ] now become tractable parameters in the model given e. We illustrate reparameterivatioo in our setting in the following example.
Example 1. Consider a dimensiimnnnoon homogeneous linear model: f(0 (x) Ox. Without loss of generality, we use only a single sample w1 from the s(w) which corresponds to the feature-independee independent temporal kernel ko l'). Again, we assume s(w) ((,
Then the time-aware hidden representation for this layer for datapoint (I1, is given by:
= (x1,1))
cos(t)w1), Ox sin(tiwi)], WI ~ N(,,
Using the reparameterization, given sample e1 from the standard normal distribution, we have: ( (41 = 1/ cos /2e1 + n)), Ozn sin (1i(0) + 1))];
(10)
so the gradients with respect all the parameters (0,, 14, 0) can be computed in the usual way.
Despite the computation advantage, the spectral density is now learnt from the data instead of being given the convergence result in Proposition does not provide cample-consisteecy guarantee. In practice, we may also misspecify the spectral distribution and bring extra intractable factors.
To provide practical guarantees, we first introduce several notations: let Kr(()) be the temporal kernel represented by random features such that KT(S) EE [ (ōT)). where the expectation is taken with respect to the data distribution and the random feature vector & has its samples {w: drawn from the spectral distribution S. Without abuse of notation, we use & ~u S to denote the dependency of the random feature vector & on the spectral distribution S provided in (8) Given a neural network
kernel (M), the neural temporal kernel then denoted by: ((/h) ((())())
(S) (h))
(S)). So the
sample version of ΣnAi (S) for the dataset (xx;+t)) is given by:
1
(S) (S)
(1)) (x;, x, (x(x To(x,,t)) & S.
(11)
n(n
If the spectral distribution S is fixed and given, then using standard techniques and Theorem D straightforward to show lim 5:(h) (S) E (h) (S)] so the proposed learning schema is sample-consistent.
In our case, the spectral distribution is learnt from the data, so we need some restrictions on the spectral distribution in order to obtain any consistency guarantee. The intuition is that if Se, does not diverge from the true S, e.g. d(So, |[S] < 8 for some divergence measure, the guarantee on S can transfer So, with the rate only suffers discount that does not depend

---

Published as a conference paper at ICLR 2021
Theorem 1. Consider the f-divergence such that d(Sor (|S) { [((dSo, /dS)ds, with the generator
function c(z) x² ffr any 0. Given the neural network kernel Σn, let M
||>>| bo then
Pr sup 1[S551
1 ( - B (So,) 2 /2exp ( 6am.
-ne?
+ +C(e),
(12)
2 mmnx ( L.
2d112 6T3
where C(E)
Sop
exp
32 max (6, M2d433))) that does not depend on
d, 2
The proof is provided in Appendix A.3.4 The key takeaway from (22 is that long as the divergence between the learnt Sa, and the true spectral distribution is bounded, we still achieve sample consistency. Therefore, instead of specifying a distribution family which is more likely to suffer from misspecification, we are motivated to employ some universal distribution approximator
such as the invertible neural network (INN) (Ardizzone et al., 2018). INN consists of series of invertible operations that transform samples from known auxiliary distribution (such as normal distribution) to arbitrarily complex distributions. The Jacobian that characterize the changes of distributions are made invertible by the INN, so the gradient flow is computationally tractable similar to the case in Example[] [
We defer the detailed discussions to Appendix AA.35 Remark 1. It clear this point that the temporal kernel approach applies to all the neural networks who have Gaussian process behavior with a valid neural network kernel, which includes the major architectures such as CNN, RNN and attention mechanism (Yang 2019).
For implementation, each forward and backward computation, we first sample from the auxiliary distribution to construct the random feature representation & using reparameterization, and then compose it with the selected hidden layer (( such as in (10). We illustrate the computation architecture in Figure 2 where adapt the vanilla RNN to the proposed framework. In Algorithm I we provide the detailed forward and backward computations, using the L-layer FFN from the previous sections as an example.
(Hidden) output at step
(Hidden) output step i+1
GRU Cell
GRU Cell
f00 0. (x, hi)
$xxx ti)
(()) (A, (Z hi.s))
(.)) cos()) ...] as in ea.(8)
sin()) cos(t) ... in (8)
(1 4+22 Fann [N 4ea] (1) o0
X1+1
t(+1
(1,(V),, 22 (V), 001m (1)), alzm((()
(WN:g(pe.)
RF-INN Module
E1.1. E2,g Esm 22.m -N(OII
Figure 2: The computation architecture of the proposed method using RNN as an example (following
Figure (p). Here, we use (O.) to denote the standard GRU cell, and use g to denote the invertible neural network. At the 7th step, the GRU cell takes the feature vector x,, and the hidden state of the previous steps húg input. The RF-INN module is called once for each batch.
4 RELATED WORK
The earliest work that discuss training continuous- time neural network dates back to LeCun et al. (1988); Pearlmutter 1995), but no feasible solution was proposed that time. The proposed approach relates to several fields that are under active research.
ODE and neural network. Certain neural architecture such the residual network has been interpreted as approximate ODE solvers (Lu et al., 2018). More direct approaches have been proposed to learn differential equations from data (Raissi & Karniadakis, 2018. Raissi et al. 2018a
7

---

Published as a conference paper at ICLR 2021
Long et al. 2018). and significant efforts have been spent on developing solvers that combine ODE and the back proppaattoitooon framework (Farrell et al. 2013] Carpenter et al. 2013 Chen et al. 2018). The closest literature to work is from Raissi et al. (2018)) who design numerical Gaussian process resulting from temporal discretization of time-dependent partial differential equations.
Random feature and kernel machine learning. In supervised learning. the kernel trick provides a powerful tool to characterize non- data representations (Shawe- Tayler et al.] 2004), but the computation complexity is overwhelming for large dataset. The random (Fourier) feature approach proposed by |Rahini & Recht (2008) provides substantial computation benefits. The existing literature on analyzing the random feature approach all assume the kernel function fixed and stationary (Yang
et al. 2012 Sutherland & Schneider 2015 Sriperumbudur & Szabó 2015 Avron et al. 2017)
Reparameterization and INN. Computing the gradient for intractable objectives using samples from auxiliary distribution dates back to the policy gradient method in reinforcement learning (Sutton al. (000). In recent years, the approach gains popularity for training generative models (Kingma &
Welling [013)) other variational objectives (Blei et al., 2017) and Bayesian neural networks (Snoek
ct al. (015)) INN are often employed to parameterize the normalizing flow that transforms a simple distribution into complex one by applying a sequence of invertible transformation functions (Dinh)
cet al.
2014 Ardizzone et al. 2018 Kingma & Dhariwal 2018 Dinh et al. 2016).
Our approach characterizes the continuous- time ODE via the lens of kernel. It complements the existing neural ODE methods which are often restricted to specific architectures, relying on ODE solvers and lacking theoretical understandings. We also propose novel deep kernel learning approach by parameterizing the spectral distribution under random feature representation, which is conceptually different from using temporal kernel for time-series classification (Li & Marlin 2015). Our work is extension of Xu et al. (2019 2020). which study the case for self-attention mechanism.
Algorithm 1: Forward pass and parameter update, using the L-layer FFN as an example.
Input: The FFN f(O..) .) (0 .)..... f(L) (0,-) }); the invertible neural network g((,,)) .); ................................
the selected hidden layer h; the loss l, associated with each input (x,,4) the auxilliary distribution P.
for each mini-batch do
Sample {El,; =2,) from the auxilliary distribution P:
Compute the reparameterized samples using the INN g(b,, e.g. wir(bb) = s(tb,
for sample in the batch do
Construct the random feature representation (x; 44) using the reparameterized
samples (so Φ is now explicitly parameterized by Φ according to eq. (8)
Forward pass: get (h) ((00,,,, x;), let (()) vb), x,, ft) == (0, x:) o (x, t4),
then pass it to the following feedforward layers to obtain the final output ÿ:
Gradient computation:
compute the gradients Vol(()) & & for the FFN and INN respectively, conditioned on the samples from the auxiliary distribution;
end
Update the parameters using the selected optimizer in a standard batch-wise fashion.
end
is straightforward from Figure 2 and Algorithm I that the proposed approach serves as plug-in module and do not modify the original network structures of the RNN and FFN.
5 EXPERIMENTS AND RESULTS
We focus on revealing the two major advantages of the proposed temporal kernel approach:
the temporal kernel approach consistently improves the performance of deep learning models, both for the general architectures such RNN, CausalCNN and attention mechanism as well as the domain-specific architectures, in the presence of continuous-time information;
the improvement is not at the cost of computation efficiency and stability, and we outperform the alternative approaches who also applies to general deep learning models.
8

---

Published as a conference paper at ICLR 2021
We point out that the neural point process and the ODE neural networks have only been shown to work for certain model architectures so we are unable to compare with them for all the settings.
Time series prediction with standard neural networks (real-data and simulation)
We conduct time series prediction task using the vanilla RNN, CausalCNN and self-attention mecha- nism with temporal kemeel approach (Figure A.I). We choose the classical Jena weather data for temperature prediction, and the Wikipedia traffic data to predict the number of visits of Wikipedia pages. Both datasets have vectorized features and are regular sample... To illustrate the advantage of leveraging the temporal information compared with using only sequential information, we first conduct the ordinary next-step prediction the regular observations, which refer as Casel. To fully illustrate our capability of handling the irregular continuous time information, we consider the two simulation setting that generate irregular continuous- sequences for prediction:
Case2. we sample irregularly from the history, i.c. xta,...,Xt,, S. k. to predict Xtzy1
Case3. use the full history to predict dynamic future point, i.e. Xtxi for a random q-
We provide the complete data description, preprocessing, and implementation Appendix We use the following two widely-adopted time-aware modifications for neural networks (denote by NN) as baselines, as well as the classical vectorized autoregression model (VAR).
NN+time: we directly concatenate the timespan, e-g. tj t, the feature vector. NN+trigo: we concatenate the learnable sine and cosine features, e.g. (sin( sin(*xt)), to the feature vector, where {m. |2= are free model parameters. We denote our temporal kernel approach by T-NN. From Figure 3 we see that the temporal kernel outperforms the baselines in all cases when the time series is irregularly sampled (Case2 and Case3), suggesting the effectiveness of the temporal kernel approach in capturing and utilizing the continuous time signals. Even for the regular Casel reported in Table A. the temporal kernel approach gives the best results, which again emphasizes the advantage of directly characterize the temporal information over discretization. We also show in the ablation studies (Appendix (B55) that INN is necessary for achieving superior performance compared with specifying distribution family. To demonstrate the stability and robustness, provide sensitivity analysis in Appendix B.6 for model selection and INN structures.
Case
Case
05
0.50
0.4
0.55
0.50
0.55
0.60
0.56
0.50
0.46
0.46
VAR
R8W
mittime
agg
T-MN
CWN
ciet-time
1-22W
Ail
sio
Agsggo
Tittt
VAR
geel tigo
1-570M
T-CWW
Air
Aigi
T-48t
Figure 3: The mean absolute error on testing data for the standard neural networks: RNN, CausalCNN (denoted by CNN) and self-attention (denoted by Att,, for the temporal kernel approach and the baselines methods in Case2 and Case3. The numerical results are averaged over five repetitions.
Temporal sequence learning with complex domain models
Now we study the performance of our temporal kernel approach for the sequential recommendation task with more complicated domain-specific two-tower architectures (Appendix B.2). Temporal information is known to be critical for understanding customer intentions, so we choose the two public e-commerce dataset from Alibaba and Walmart.com, and examine the next-purchase recom- mendation. To illustrate our flexibility, we select the GRU-based, CNN-based and attentionnbbsee recommendation models from the recommender system domain (Hidasi et al. 2015 Li et al., (017a) and equip them with the temporal kernel. The detailed settings, ablation studies and sensitivity analy- sis are all in Appendix B The results are shown in Table A.2 We observe that the temporal kernel approach brings various degrees of improvements the recommendation models by characterizing

---

Published as a conference paper at ICLR 2021
the continuous-ting time information. The positives results from the recommendation task also suggests the potential of our approach for making impact in boarder domains.
DISCUSSION
In this paper, we discuss the insufficiency of existing work on characterizing continuous- data with deep learning models and descrihe principled temporal kernel approach that expands neural networks to characterize continuous-time data. The proposed learning approach has strong theoretical guarantees, and can be easily adapted broad range of applications such deep modelling, outlier and burst detection, and generative modelling for time series data.
Scope and limitation. Although the temporal kernel approach is motivated by the limiting- width Gaussian behavior of neural networks, in practice, it suffices to use regular widths as we did in our experiments (see Appendix BB. for the configurations). Therefore, there are still gaps between our theoretical understandings and the observed empirical performance. which require more dedicated analysis. One possible direction is to apply the techniques in Daniely et al. (2016) to characterize the dual kernel view of finite-wids neural networks. The technical detail, however, will be more involved.
is also arguably true that we build the connection between the temporal kernel view and continuous- time system in an indirect fashion, compared with the ODE neural networks. However, our approach is fully compatible with the deep learning subroutines while the end-to-end ODE neural networks require substantial modifications to the modelling and implementation. Nevertheless, ODE neural networks are (in theory) capable of modelling more complex systems where the continuous time setting is special case. Our work, the other hand, is dedicated the temporal setting.
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. In Advances in neural information processing systems, pp. 6155-6166, 2019.
Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W Pellegrini, Ralf Klessen, Lena Maier-Hein, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. arXiv preprint arXiv:1808. 04730, 2018.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances Neural Information Processing Systems, pp. 8139-8148, 2019a.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. arXiv preprint arXiv:1901.08584 2019b.
Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandieh. Random fourier features for kernel ridge regression: Approximation bounds and statistical guarantees. In International Conference on Machine Learning, pp. 253-262, 2017.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint 100.:v:14090473: 2014.
Atilim Günes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. The Journal of Machine Learning Research, 18(1):5595-563 2017.
Aharon Ben-Tal, Dick Den Hertog, Anja De Wacgenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59())331-3357, 2013.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112511))859-877, 2017.
Salomon Bochner. Vorlesungen über Fouriersche integrale. Chelsea Publishing Company, 1948.
10

---

Published as a conference paper at ICLR 2021
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
Peter J Brockwell, Richard A Davis, and Stephen E Fienberg. Time series: theory and methods: theory and methods. Springer Science & Business Media, 1991.
Bob Carpenter, Matthew D Hoffman, Mareus Brubaker, Daniel Lee, Peter Li, and Michael Betan- court. The stan math library: Reverse-mode automatic differentiation in c++. arXiv preprint arXiv:1509.07 164, 2015.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in neural information processing systems, pp. 6571-6558 2018.
Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter Stewart, and Jimeng Sun. Doctor ai: Predicting clinical events via recurrent neural networks. In Machine Learning for Healthcare Conference, pp. 301-318, 2016.
Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of initialization and dual view on expressivity. In Advances In Newral Information Processing Systems, pp. 2253-2261, 2016.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.
Laurent Dinh, Jascha bohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803 2016.
Arnaud Doucet and Adam M Johansen. tutorial on particle filtering and smoothing: Fifteen years later. Handbook of nonlinear filtering, 12(656-704):3, 2009.
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Nomez-Rodriguck, and Le Song. Recurrent marked temporal point processes: Embedding event history vector. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp- 1555-1564, 2016.
Patrick E Farrell, David A Ham, Simon W Funke, and Marie E Rognes, Automated derivation of the adjoint of high-level transient finite element programs. SIAM Journal Scientific Computing, 35 (4):C369-C395 2013.
Joseph Futoma, Sanjay Hariharan, and Katherine Heller. Learning detect sepsis with a multitask gaussian process mn classifier. arXiv preprint 47Xiv17706004155 2017.
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.066 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98): 1735-1780, 1997.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernll: Convergence and generalization in neural networks. In Advances in neural information processing systems, pp. 8571-8580, 2018
Diederik P Kingma and Max Welling. 1rXiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018
Yann LeCun, D Touresky, G Hinton, and T Sejnowski. A theoretical framework for pock-propagation. In Proceedings of the 1988 conmectionist models summer school, volume 1, pp. 21-28, CMU, Pittsburgh, Pa: Morgan Kaufmann, 1988.
11

---

Published as a conference paper at ICLR 2021
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in neural information processing systems, pp. 8570-8581, 2019.
Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. Neural attentive session- based recommendation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 1419-1428, 2017a.
Shuang Li, Shuai Xiao, Shixiang Zhu, Nan Du, Yao Xie, and Le Song. Learning temporal point processes via reinforcement learning. Advances in neural information processing systems, pp. 10781-10791, 2018.
Steven Cheng-Xias Xian Li and Benjamin M Marlin. Classification of sparse and irregularly sampled time series with mixtures of expected gaussian kernels and random features. In UAI, pp. 484 493, 2015.
Yang Li, Nan Du, and Samy Bengio. Time-dependent representation for neural event sequence prediction, arXiv preprint arXiv: 00065, 2017b.
Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with rnns: Improved classification of clinical time series. In Machine Learning for Healthcare Conference, pp. 253 270, 2016.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. In International Conference Machine Learning, pp. 3208-3216, 2018.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pp. 3276-3285. PMLR, 2018
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. In Advances in Neural Information Processing Systems, pp. 6754 46664, 2017.
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association, 2010.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Roman Novak, Lechao Xiao, Jachoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. arXiv preprint arXiv:1810.05144 2018
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.034499 2016.
Barak A Pearlmutter. Gradient calculations for dynamic recurrent neural networks: A survey. IEEE Transactions Neural networks, 6(5):1212-1228, 1995.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, pp. 1177-1184, 2008
Maziar Raissi and George Em Karniadakis. Hidden physics models: Machine learning of nonlinear partial differential equations. Journal of Computational Physics, 357:125- 1411 2018
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Multistep neural networks for data-
driven discovery of nonlinear dynamical systems. arXiv preprint arXiv:1801.01236. 2018a.
Maziar Raissi, Paris Perdikaris, and George Em Kamiadakiss. Numerical gaussian processes for time-dependent and nonlinear partial differential equations. SIAM Journal on Scientific Computing. 40(1):A122-A195 2018b.
12

---

Published as a conference paper at ICLR 2021
Sami Remes, Markus Heinonen, and Samuel Kaski. Non-stationary spectral kernels. In Advances in Neural Information Processing Systems, pp. 4642-4651, 2017.
Simo Särkkä. Biterinn and smoothing, volume 3. Cambridge University Press, 2013.
John Shawe-Taylor, Nello Cristianini, et al. Kernel methods for pattern analysis. Cambridge university press, 2004.
Mikhail Aleksandrovich Shubin. Pseudodiiffrrenicl operators and spectral theory, volume 200. Springer, 1987.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In International conference on machine learning, pp. 2171-2180 2015.
Hossein Soleimani, James Hensman, and Suchi Saria. Scalable joint models for reliable uncertainty- aware event prediction. IEEE transactions on pattern analysis and machine intelligence, 40(8): 1948-1963, 2017.
Bharath Sriperumbudur and Zoltán Szabó. Optimal rates for random fourier features. In Advances in Neural Information Processing Systems, pp. 1 1144-1 1152, 2015.
Dougal Sutherland and Jeff Schneider. On the error of random fourier features. arXiv preprint 1rXiv:1506.02785 2015.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Advances neural information processing systems, pp. 1057-1063, 2000.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Norbert Wiener al. Generalized harmonic analysis. Acta mathematica, 55:117-258 1930.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378, 2016.
Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Self-attention with functional time representation learning. Advances in Neural Information Processing Systems, pp. 15889-15880 11899, 2019.
Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Inductive representa- tion learning on temporal graphs. arXiv preprint arXiv:2002.07962. 2020.
Akira Moiseevich Yaglom. Correlation theory of stationary and related random functions. Volume I: Basic Results., 526, 1987.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint 17Xiv.1902.04760. 2019.
Tianbao Yang, Yu-Feng Li, Mehrdad Mahdavi, Rong Jin, and Zhi-Hua Zhou. Nyström method vs random fourier features: A theoretical and empirical comparison. In Advances neural information processing systems, pp. 477,488, 2012.
13

---

Published as a conference paper at ICLR 2021
A
APPENDIX
We provide the omitted proofs, detailed discussions, extensions and complete numerical results.
A.1
NUMERICAL RESULTS FOR SECTION
T-GRU
T-(CausaIJCNN
T-(SelFAAttention
RFF-INN
output
(g)
Wg
Transpose
Attention
0%g Fg)
(gg
casual mask
weight
X4
ERU coll
(g
NNT-88
(3..5)
Wii-Hb +8
softmax
A4w
(5..44)
outout
(h,(t.)
y4(t))
Eg)
Wy
GRU cell
(g
Figure A.I: Visual illustrations how we equip the standard neural architectures with the temporal kernel using the random Fourier feature with invertible neural network (the RFF-INN blocks).
Case 1
Weather
Wikipedia
VAR
0.2643
2.31
RNN
0.24877.003
0.51427.003
RNN-time
0.2629/.001
0.4698/.00
R.N--rrio
0.2526/.003
0.4542/.004
T-RNN
0.2386/0002
0,4330/.002
CNN
0.31037.000 000
0,49987 003
CNN-time
0.2933/.003
0.4852/.001
CNN-trigo
0,2684/.004
0.4556/.002
T-CNN 0.2662/003
0,4399/003
Aftention
0.40527.003
0.47957.000 000
Attention-time
0.4298/0003
0.4809/
Autention-trioo
0.2887/002
0.4445/.003
T-Attention
0.2674/.004
0.4226/.002
Table A.1: Mean absolute error for time series prediction of the regular scenario of Case We underline the best results for each neural architecture, and the overall best results are highlighted in bold-font. The reported results are averaged over five repetitions, with the standard errors provided.
Alibaba
Walmart
Metric
Accuracy
DCG
Accuracy
DCG
GRU-Rec
77.817.1
47.12/.11
18.09/.13
3.44/.21
GRU-Rec-lime
77.70/.24
46.21/.13
17.66/.16
3.29/.23
GRU-Rec-trio
78.95/.19
49.01/111
21.54/.13
6.67/.18
T-GRU-Ree
79,471.35
49,82/.40
23.41/.11
8.44/21
CNN-Rec
74.89/.33
43.91/222
15.98/.18
1.97/.19
ONN-Rec-time
74.85/.31
43.88/.21
15.95/.18
1.96/11
CNN-Rec-trigo
75.97/.21
45.86/.23
17.74/.17
3.80/.15
T-CNN-Rec
16.45/.16
46.55/.38
18,59/33
4.56/.31
ATTN-Rec
51.82//44
30.417.52
20.41/.88
7.52/118
ATTN-Rec-time
51.84//43
30.45/.50
20.43/.36
7.541.19
ATTN-Rec-trigo
53.05/.30
33.10/29
24.491.15
8,93/113
T-ATTN-Rec
53,49/.31
33.58/.0
25.51/.17
9.22/.15
Table A.2: Accuracy and discounted cumulative gain (DCG) for the domain-specifer models on the temporal recommendation tasks. See Appendix (B for detail. We underline the best results for each neural architecture, and the overall best results are highlighted bold-font.
14

---

Published as a conference paper at ICLR 2021
A.2 SUPPLEMENTARY MATERIAL FOR SECTION 2
We discuss the detailed background for the Gaussian process behavior of neural network and the training trajectory under neural tangent kernel, well the proof for Claim [
A.2.1 GAUSSIAN PROCESS BEHAVIOR AND NEURAL TANGENT KERNEL FOR DEEP LEARNING MODELS
The Gaussian process (GP) view of neural networks at random initialization was originally discussed in (Neal, 2012). Recently, CNN and other standard neural architectures have all been recognized as functions drawn from GP in the limit of infinite network width (Novak et al. 2018) Yang 2019). When trained by gradient descent under infinitesimal step schedule, the gradient flow of the standard neural architectures can be described by the notion of Neural Tangent Kernel (NTK) whose asymptotic
behavior under infinite network width is known (Jacot et al. 2018). The discovery of NTK has led to several papers studying the training and generalization properties of neural networks (Allen-Zhu al. 2019 Arora et al., 2).
For L-layer FNN f(0, x) = f(L) with hidden dimensions {dA Vi=1 and recursively defined via: p(L) = -(1)) (x) + b(0), ((h)(x) 1 w( (f + b ((0) = x, ddn
(A.1)
for h = 1, 1,2, ..... L - 1, where (.) is the activation function and the layer weights W(1) E R 1, ((h) E IRd xda and intercepts are initialized by sampling independently from No0, 1) (without loss of generality). As di....,de di + co. (() tend in law to i.i.d Gaussian processes with covariance E defined recursively as shown by Neal (002)):
1 XTx +1, E (x, x') N1, ((())))))))((((())))(((())))))) +1. (A.2)
2) (x,
h1
We also refer to ((h) as the neural network kernel to distinguish from the other kernel notions. Given training dataset (x,, Vo 1 let (0(s)) = ((((s)())))).....)))))))) f(((s,, xx)) be the network outputs the th training step and y (y1.... (y1....,,.)) Vn))
When training the network by minimizing the squared loss f(0) with infinitesimal learning rate, i.c. do(a) ()))())) the network outputs training step & follows the evolution Jacot et al. 2018):
de
df (f(s)) - ) x (f()) y), [0(s) = ( 3 (f(((ss, 2x) 0f (O)))
(A.3)
ds
00
00
The above (s) is referred to as the NTK. and recent results shows that when the network widths go to infinity (or sufficiently large). (s) converges to fixed Ou almost surely (or with high probability).
For standard L-layer FFN, the NTK O ( for parameters (W()), bud) on the h layer can also be computed recursively:
(x., = 2)(h) (x,,x)
(x,, x²) Ey~Nio, (f
(A.4)
and θ (x,, x2) et (x, x,) ) (x, x + (()) (x,, xx), k= h
A number of optimization and generalization properties of neural networks be studied using NTK, which we refer the interested readers to (Lee et al. 2019 Allen-Zhu et al., 2019 Arora et al. 2019a b). We also point out that the above GP and NTK constructions can be carried out on all standard neural architectures including CNN, RNN and the attention mechanism Yang 2019)
A.2.2 PROOF FOR CLAIM 1
In this part, we denote the continuous-timg system by X(t) in order to introduce the full set notations that are needed for proof, where the length of the discretized interval is explicitly considered. Note that we especially construct the example in Section so that the derivations are not too cumbersome. However, the techniques that use here can be extended to prove the more complicated settings.
15

---

Published as a conference paper at ICLR 2021
Proof. Consider (t) to be the second-order continuous-timi time autoregressive process with covariance function k(t) and spectral density function (SDF) s(w) such that s(w) = f- exp( - ind (t)))
The covariance function of the discretization Xa[n] X ((n) with any fixed interval > 0 is then given by ha in] = k(na). According to standard results in time series, the SDF of (t) is given by in the form of:
s s(w)
+ + = 0, a + azbY + 0.
(A.5)
w bj
W + b2
We assume without loss of generality that b,, are positive numbers. Note that the kernel function for Xa[i] can also be given by
Ka [n]
(sp(ianw)s(()))))
12x+1)x
exp(inw)s
(A.6)
(2k- 1)mt
k=-
1
Σ s( w +
2k
exp(inw)
dw.
Am--
which suggests that the SDF for the discretization X² [n]] can be given by:
a Σ W + 2km
sa(w)
2ab, -
2abz - 1
(1
(A.7)
2
bi leab clu12
b1 cabz cdu
a1 (dg - 2d2 cos(w))
2b, bz (e e(w)( (eabz e/w))22
where d2 = bbe (e2abz 1) - by eab (e (c - 1). By the definition of discrete- time auto-regressive process, X.[ni is second-order AR process only if d2 O, which happens if and only if: by/b1 (cnbz - e e-bb)) //(()) - e e-ab)), However, the function g = cxp(az) - exp(-axt is concave on 00) (since the time interval
0) and (0) = the-above equality hold if bj b2. However, this contradicts with (A.5), since ai + 02 = 0 and a, bz + a2l2 * 0 suggests as (b) - b2)² + 0.
Hence, X. [n] cannot be a second-order discrete-time auto-regressive process.
A.3 SUPPLEMENTARY MATERIAL FOR SECTION
We first present the related discussions and proof for Claim 2 on the connection between continuous- time system and temporal kernel. Then we prove the convergence result Theorem I regardinn the random feature representation for non-stationary kernel. In the sequel, we show the new results for the Gaussian process behavior and neural tangent kernel under random feature representations, and discuss the potential usage of our results. Finally, we prove the sample-consistency result when the spectral distribution is misspecified.
A.3.1 PROOF AND DISCUSSIONS FOR CLAIM
Proof. Recall that we study the dynamic system given by:
an (x) d" f(x,t) t)
+ +... ax(xx)(xx)) = bm (x) d exx,(xx t) + .... boe(x, !),
(A.8)
d
dmm
where <(x, /. = to) ~ NO., ) Vto T. The solution process to the above continuous-time system also Gaussian process, since <(x, f) is Gaussian process and the solution of linear different equation is a linear operation the input. For the sake of notation, and by (x) = O,... bm (x) = O, which does not change the arguments in the proof. We apply the Fourier transformation transformation on both sides and solve the for Fourier transform fliwx, iw):
assume bo = 1
1
fllwxx iw) (x) (bw)9 + (x) is + oo(x) W (dw) wx),
(A.9)
16

---

-wr). Since p(ax, w) is now in rational function form Published as a conference paper at ICLR 2021
where W(ws wx) is the Fourier transform of e(x.t). If do not make the assumption (b((x))]== they will simply show up on the numeration in the same fashion as (x)(x) Let
Ge (iw; x) (x(x). (iw)) +... a1 (x) is + ao(x),
and p(w>)) [iwr; 22) be the spectral density of the Gaussian process corresponding c (its spectral density does not depend on because is a Gaussian white noise process the time dimension). The dependency of G(-;)) on Or because we defined Or to the underlying parameterization of {ay (=)) in the statement of Claim 2 Then the spectral density of the process
fx, t) is given by
p( C p(wx))Gor (ww; x)1² x p(wx)por (w;
where C' is constant that corresponds to the spectral density of the random Gaussian noise on the time dimension. Notice that the spectral density function obtained this way is regular, since it has the form of por (w: x) = constant/ (polyyomiall of w²)
Therefore, according to the classical Khener-Khinchin theorem Brockwell et al. (199T). the covariance function of the solution process is given by the inverse Fourier transform of the spectral density:
2x / plw wxx exp (ilw, wx) [1, x)) ()))))(((ss wx)
(x,
x / por (w: x) exp(iiotdds f pl(wx) explii]x )dwx
(A.10)
x Ker ((x, t), (x, E(h) (x,
And therefore we reach the conclusion in Claim 2 by taking >: (x, t; x', v(x x,
The inverse statement of Claim 2 may not be always true, since not all the neural-temports kernel can find a exact corresponding continuous-time system in the form (A.8). However, we may construct the continuous-time system that approximates the kernel (arbitrarily well) in the following way, using the polynomial approximation tools such as the Taylor expansion.
For a neural- -emporlll kernel I(h)
LT', we first compute it Fourier transform to obtain the spec- tral density p(wx.(), Note that p(wxx w) should be rational function in the form of (polynomial in / in w²), or otherwise it does not have stable spectral factorization that leads to linear dynamic system. To achieve the goal, we can always apply Taylor expansion or Pade approximants that recovers the p(wx, w) arbitrarily well.
Then we conduct a spectral factorization on w)) to find G(iwx, iax) and p(wx) such that plwy, w) = G(iwx, (w: (apppwx)G((iiee of wa, we can find G(iwxx iwr) as:
by (iwx) (iw) 8 + ... by (iwx): (zw) + (d(()x))
ag (iwx) (iw) + + ai (iwx): (tw) + ag (iwxx)
Let ay (x) and 8,(xx be the geeudoddifferntia. operators of a (1wس)) and by (1we)) defined in terms of their inverse Fourier transforms (Shubin 1987), then the corresponding continuous-time system is given by:
d [xx,t) t) +... (x) ag( (x)
d'e(tt) dt
(x(x))(t).
(A.11)
dta
For concrete end-to-end example, we consider the simplified setting where the temporal kernel function is given by:
Ko, (t, E') = ko, t') -0)==1(((0)) (V281
21-0.
2/281'-1'00 Bo, (V20g 12))
r(()))
0g
0g
where Bg, (+) is the Bessel function Ko, (t, belongs the well-known Matern family. It is straightforward to show that the spectral density function is given by:
20
+ w2 -(01-1/2)
s(w) α
0
17

---

Published as a conference paper at ICLR 2021
/220 -(0++1/2)
(V201
-(1++1/2)
As consequence, that s(w) x
nw)
so
0g -11++1/2)
03
(V20 03
directly have Go, (w)
+ kw
instead of having to seek for polynomial approxi-
mation. Now easily expand (w) using the binomial formula to find the linear parameters for the continuous-time time system. For instance, when 01 3/2, we have:
d' f(t) d/2
201 df +
201 f(t) c(t).
+
03
dt
0g
A.3.2 PROOF FOR PROPOSITION
1
Proof. We first need to show that the random Fourier features for the non-stationary kernel Kr( ((x, 1), (x)) can be given by (11), i.e.
1 [ ..... cos (Ix, 1).) + cos ([x, (T2.:) sin ([x, t| (w.,)) :) +sin ([x, tTT,.) ...]. Φ(x, t) 2/m
To simplify notations, we let z == |x, t Rd+1 and Z X' x T. For non-stationary kernels, their corresponding Fourier transform can be characterized by the following lemma. Assume without loss of generality that Kr is differentiable.
Lemma A.1 Yaglom 1987 ). A non-stationary kernel k:22, 22) is positive definite in Bdd if and only if after scaling, it has the form:
k(z,, z2) = fexp
f exp(i((Tzz wh (( dwz):
(A.12)
where u(dw), dwz)) is some positive-semideffiie probability measure with bounded variation.
The above lemma can be think of as the extension of the classical Bochner's theorem underlies the random Fourier feature for stationary kernels. Notice that when covariance function for the measure only has non-zero diagonal elements and w1 = w2, then we recover the spectral representation stated in the Bochner's 's theorem. Therefore, However, we need to ensure the seattive-semidefiittneess of the spectral density for p(dw), dwz)), which we denote by p(w1, It has been suggested in Remes al. (2017) that we consider another density function g(w), w2) and let p be taken on the product space of q and then symmetrise:
can also approximate (A.12) with the Monte Carlo integral.
p(w1. (q(w), w2) + g(w2, w1) + ((w).w1) + ((w2, w2))
(A.13)
Then (.12 suggests that
1
k(21, 22) = 1 Bny exp (ilw + exp (ilw)z2 - w/Z1))
exp (ilwiz) - wf21)) + exp (ilw)z2 - wfzza)]
wjzz). So with the Trigonometric equalities, it is straightforward to verify that k(z1, 22) = F, ( Hence, the random Fourier features for non-stationary kernel can be given in the form of (11).
Recall that the real part of exp (i(w]z) w3zz)) is given by cos(w
Then we show the uniform convergence result as the number of samples goes to infinity when comput- ing E.g (x(()) T<(2)] by the Monte Carlo integral. Let 2 Z Z, so 2 = {(x, (xx,,,x4,,) x, x' X; t E T). Since diam(X') = I and T (max)] we have diam 2) = [2/2 Let the approximation error be
A(z,z') T - (z.zz').
(A.14)
The strategy is to use e-net covering for the input space 2, which would require N (2/2 /r) d-1 balls of radius Let = [c be the centers for each e-ball. We first show the bound for (cc;)] and the Lipschitz constant La of the enror function A, and then combine them to get the desired result.
18

---

Published as a conference paper at ICLR 2021
Since A is continuous and differentiable w.I.t z' according to the definition of ó. we have LA \VA(c")l: where c* = arg maXcgc / Let c* = (2. z'). By checking the regularity conditions for exchanging the integral and differential operation, we verify that IE ([( = (((2')) VI (z, 2']]
We do not present the details here, since it is easy to check the regularity of o ((z))Tl (z') as it consists of the sine and cosine functions who are continuous, bounded and have continuous bounded derivatives. Hence, we have:
[13]
Eg,g
[V(z) V
Eg
(V (()') + V (2, 22
Ex,
(by Jensen's inequality)
I (
E
V cos( 2 w1 + cos wZ cos (os((())) (z'))=w)) + cos((z')Tuu))
+ (sn(( sin(z w2)) sin((z')\au) + sin sin((zz)Tw>)) ||
2R wi (wi (z)) + sin (z) w2 ااس))
+ w2 sin( w (Z)) sin((E)) Tw2 TTwz2)
< <8B||W<, wا]| 80
(A.15)
Hence, by the Markov's inequality, we have
p La
2 (
) 80, s8,; (2))
s
(A.16)
Then we notice that for all c E C, A(c) is the mean of m/2 terms bounded by [-1, 1], and the expectation O. So applying a union bound and the Hoeffding's 's inequality on bounded random variables, we have:
J |a()) 2 :) S 2N exp (-
me²
(A.17)
16
Combining the above results, we get
3203,² 2²
2/2 t
d+1
me²
p sup (xC
A(zzz')) SE) 21
2r
(d+11]
exp
16
(A.18)
2(d+1)/(d+3)
me²
2 C(d)
exp
c
8(d + 3)
(d+1)k -(att)
1/(d+3)
where in the second inequality we optimize over r such that r"
with
2 maxx
) dd+1 exp( (xp(-mee/16) and kn = 3203c-2 The constant term is given by C(d)
=
2
7a19
[+1
21 + गंड
A.3.3 THE GAUSSIAN PROCESS BEHAVIOR AND NEURAL TANGENT KERNEL AFTER COMPOSING WITH TEMPORAL KERNEL WITH THE RANDOM FEATURE REPRESENTATION
This section is dedicated to show the infinite-wiee width Gaussian process behavior and neural tangent kernel properties, similar to what we discussed in Appendix A.2 when composing neural networks in the feature space with the random feature representation of the temporal kernel.
For brevity, we still consider the standard L-layer FFN of (A.I) Suppose we compose the FFN with the random feature representation ofx xx 1) at the t layer. It is easy to see that the neural network
19

---

Published as a conference paper at ICLR 2021
kernel for the first k 1 layer unchanged, so compute them in the usual way as in (A.2) For the ttth layer, it is straightforward to verify that:
lim E 1 du 400 dg (w(h)
(0 ((-
(0, o t(x,t), w(k) W(k) ((h-1) (0, x') o (o(xxx)) ((h-1)
+x Kr (xx t), (x',
The intuition is that the randomness in W (thus f(O..)) and ((...) are independent, i.e. the former caused by network parameter initializations and the later is induced by the random features. The covariance functions for the subsequent layers can be derived by induction, e.g. for the (k + 1)th layer we have:
E (1+1) (x, E ( (F) (0
In summary, composing the FNN, at any given layer, with the temporal kernel using its random feature representation does not change the infinite-widhe width Gaussian process behavior. The statement true for all the deep learning models who also have the Gaussian process behavior, which includes most of the standard neural architectures including RNN, CNN and attention mechanism (Yang) 2019).
The derivations for the NTK, however, more involved since the gradient on all the layers are affected. We summarize the result for the L-layer FFN in the following proposition and provide the derivations afterwards.
Proposition A.1. Suppose (k) (0, (x, = (K) x) Φ(x, 1)) in the standard L-layer FFN. (h) (h) for h 1,..., 2/(0 >1(h)) KT and E =
(o (f)) + for h = +l+1.... L.. If the activation functions have polynomially bounded weak derivatives, as the network widths d!,...,d,, - 0C, the neural tangent kernel (L) converges almost surely to (L) whose partial application on parameters {W(N), ((b) in the 1tth -laye is given recursively by:
Let 2T
T
r = -
(h) θ (k)
(1) k=
(A.19)
T
Proof. The strategies for deriving the NTK and show the convergence has been diseussed in Jacot et al. (2018) Yang 2019); Arora et al. (2019a). The key purpose for us presenting the derivations here is to show how the convergence results for the neural-temporal Gaussian (Section 4.2) affects the NTK. To avoid the cumbersome notations induced by the peripheral intercept terms, here we omit
the intercept terms b in the FFN without loss of generality. We let g(k) Va, a (f(h) 1)), so the
FFN can be equivalently defined via the recursion:
(A) = (h()) g ((a-1) (x,t) t). For the final output f(0, (xxt)) == ((.) (()) (x, t), the partial derivative to (hh) can be given by:
as ( (x,()) 20hh (h-1))11))))) (x
(A.20)
(h())
with (b)) defined by:
z (h) (x. t) = = D (x, (x, (Wih+))22(00)))
h T.,
(A.21)
h 1, ..... L 1,
vil
where
D (x,()) diag
à (x)
= k,.,...... To
diag à (((()()))) h I..nk, k- 1.
Using the above definitions, we have: as(o,
(x,)) (x,1) as(0, ((,,(xx)))) awm) aw
(xx (x- (x, (x = =ggh (1 (x) (1- (x){ (x) 2x)
20

---

Published as a conference paper at ICLR 2021
We have established in Section 4.2 that
(g hh 11 (x, t), g) (x, + 2 ((x, (x,,)),
where
h 1, ......
204h ((x, (x,,))
(x, Kr (x, (x,,
h = k
E g~ (0. 1 // -1)
0 (f (f
h = k + 1, L.
(A.22)
By the definition of (h),, we get
(z (x. t), 2 (1) (1) {)) 1 (D (x, ( (1+1)) 8), (x (w(h (1+11 (x (t)
dn
(D( (xx ((++)) ( t). D (1) (x' (' (W 11 2
(x( 11)
(A.23)
dn.
1
tr D (b) (x, (h) (x, (1) (2) (2( (x, 1), x (h- 11 (x'
dp
+ 1), (x', (2 (h+ +1) (x, t), z (1+ (x, t'))
The approximation in the third line is made because the (h++1 in the right half is replaced by its i..d copy under Gaussian initialization. This does not change the limit when dy + oo when the actionvation functions have polynomially bounded weak derivatives Yang (2019) such the ReLU activation. Carrying out (A23) recursively, we see that
L-1 & ((,,),
(1) (x,,)) - II (2 ()x,1),2 2))
j=h
Finally, we have:
05 (x,1) 0f(0((x',1)
05 (x,
00(0((x11))
Σ
00
00
@w(h)
ow())
(A.24)
1
Σ
14 ((x,t)
II
Đ ((x)),
Notice that we use a more compact recursive formulation to state the results in Proposition 1. is easy to verify that after expansion, reach the desired results.
Compared with the original NTK before composing with the temporal kernel (given by (A.4)) the results in Proposition A. shares a similar recursion structure. As consequence, the previous results for NTK can be directly adopted to setting. We list two examples here.
Following Jacot et al. 2018 given training dataset (xi, li, (:(4)) let fr (0())) = (f() X1, 11) yr (y) (t1).... ..... yn (t,)) The analysis the optimization trajectory under infinitesimal learning rate can be conducted via:
f (s(s), Xn, tn)) be the network outputs at the 8th training step and
(f(( = Or((s) x (fr()(s) -r),, ds
where Or(s) converges almost surely to the NTK θ (L)) in Proposition A.I
Following Allen-Zhn et al. (2019) and Arora et al. (001bb). the generalization performance of the composed time-aware neural network can be explicitly characterized according to the properties of (L)
21

---

Published as a conference paper at ICLR 2021
A.3.4 PROOF FOR THEOREM 1
Proof. We first present a technical lemma that is crucial for establishing the duality result under the distributional constraint (S [S] S 6. Recall that the hidden dimension for the 4 layer is dg-
Lemma A.2 (Ben- Tal al. (2013). Let f be any closed convex function with domain [0, + and this conjugate is given by [s)s = supizo f(t)}. Then for any distribution S and any function
g Rd++1 R, we have
/ g(w)dSo,
{s fr
( ( A 7) ds(w) + 82 +n). (A.25)
sup Soy Segidy(So, [|5)
= inf x20.0
We work with scaled version of the I-divergence under f(t) (tt 1) (because its dual function has a cleaner form), where the constraint set is now equivalent to (So, (S ||5] I 6/k}. f. It is easy to check that (s) = [ + & with F + 1.
Similar to the proof for Proposition I, we let z = [x, t] E [R++1 and Z = X' x T to simplify the notations. To explicitly annotate the dependency of the random Fourier features on 52, which the random variable corresponding to w, we define ő(z, 52) such that क(z, 52) = cos(2 - cos(zT(2), R2), sin() + [22)], where Ω = [22]. Then the approximation error, when replacing the sampled Fourier features & by the original random variable o(z, Ω), is given by:
A, (2)) = nnnn 1)
1
Σ
x(5) (xi, X3 (2,, 22) (22
-E (X, x, Kr Kr,50, ((x,, (X),)(X,,)) Tj))]
1
(x) (x,,,))0, 22) To(z, 22) - ( (X, (xX)))( .2)].
nn
(A.26)
We first show that sub-Gaussianity of A, (2) Let {x be an i.i.d copy of the observations except for one clement j such that X, * X,. Without loss of generality, we assume the last element different, i.e. Xn o x, Let A (S2) be computed by replacing x and with the above and its corresponding z'. Note that
(12) (9))
[k)) (1)x,
(4 () ?2)
n(n - 1)
$ n(n - 1)
X x. )o( 12) 22) Ix x2)( (A.27)
ien
+ Σ
X, ) (x,)
22) 2) (x)) (00,,,,,,)
j<n
4max (1,
n
where the last inequality comes from the fact that the random Fourier features 3 are bounded by 1 and the infinity norm of (() is bounded by M. The above bounded difference property suggests that (2)) is 1 (nax(1.M) Saub-Gaussian random variable.
n
22

---

Published as a conference paper at ICLR 2021
To bound (S2), use:
/ (s))ddSo
(
sup Soy (S0 (18)
sup
Soy (S,
{
1(5 + 1) Es (52))
{ (using Lemma 2)
S inf
A20
Wv'
&
(6 1) |A,, (52)
1/K (solving for X from above)
vo + LEs |IA. (12) 1/2 (Iet l K' 1/2).
(A.28)
Therefore, to bound supsey (Sy [|5)
/ A, ( we simply need to bound [ (2)) Us- (Boucheron et al. 2013). for À S n/ we
ing the classical results for sub-Gaussian random variables have
exx (2), I exp (- 1
log(1 - 8 M1)
2
Then we take the integral over w
pl / An(w)) 8 5+1
- exp () An(w) exp (-
Ac2
8 8+11 8+1)
(Chernoff bound)
(A.29)
8 mmax L, ( M3..
Sexp ( log
(apply Jensen's inequality).
8 5+11
n
Finally, the true approximation error (.( (So)) -x(4) Notice that
1
(n) < (2))
n(n Σ (x,, (0 22) TZ(z !!) (2x) (ZZz))
From A.28) and A.29 we are able to bound supse, .dy (Se, (15) 4n (2). For the second term, recall from Proposition that we have shown the stochastic uniform convergence bound for 1 2, 2) (l2 22) T((,)) under any distributions So,, The desired bound for
sup (Se, (1S) (n( 2 2c) is obtained after combining all the above results.
A.3.5 REPARAMETRIZATION WITH INVERTIBLE NEURAL NETWORK
In this part, diseuss the idea of constructing and sampling from arbitrarily complex distribution from known auxiliary distribution by a sequence of invertible transformations. Given an auxiliary random variable following some know distribution g(2), suppose another random variable is constructed via one-to-one mapping from z: fzz), then the density function of x is given by:
dz dx
p(x) <(>)
{s 1xx))
(A.30)
dx
We parameterize the one-to-ona function f(-) with free parameters 0 and optimize them over the observed evidence such as by maximizing the log-likelihood. By stacking sequence of Q one-to-one mappings, i.e. fQ JQ-1 ... fi (z), can construct complicated density functions. It is easy to show by chaining that p(x) is given by:
log p(x) log q(z)
(A.31)
dz,
Samples from the auxiliary distribution can be transformed to the unknown target distribution in the same manner, and the transformed samples are essentially parameterized by the transformation mappings.
23

---

Published as a conference paper at ICLR 2021
Unfortunately, most standard neural architectures non-invertible, settle specific family of neural networks the invertible neural network (INN) Ardizzone et al. (2018). A major component of INN is the affine coupling layer. With sampled from the auxiliary distribution, first divide into two halves [1, ø2] and then let:
[
V1 = 21 ④ exp (1) (7, z2)) tı (7, z2)
(A.32)
V2 = 22 o exp (82(7, 21)) (2 21),
where 8i (1,
t (7, 1)) can be any function parameterized by different parts of 7. Here, denotes the element-wise product. Then the outcome is simply given by:
ti (7, (7,
g(y, z) = [V1 V2l.
To see that g(7,) ) is invertible so the inverse transform mappings are tractable, it is straightforward show that g (r, [Vi, v2]) is given by:
Z2 = (v2 - v1(7,v1)) @ exp - (1, v1))
(A.33)
21 (vi v2( exp (- (2)2
By stacking multiple affine coupling layers, scalar multiplication and summation actions (which are all invertible), we are able to construct an INN with enough complexity to characterize any non-degenerating distribution.
B SUPPLEMENTARY MATERIAL FOR SECTION 5
We provide the detailed dataset description, experiment setup, model configuration, parameter tuning. training procedure, validation, testing, sensitivity analysis and model analysis. The reported results are averaged over five iterations.
B. DATASETS
Jena weather datase The dataset contains 14 different features such as air temperature, atmospheric pressure, humidity, and other metrics that reflect certain aspect of the weather. The data were collected between between 2009 and 2016 for every 10 minutes, so there 6 observations each hour.
A standard task on this dataset is to use days of observations to predict the temperature 12 hours in the future, which we refer as the Case We use sliding window obtain the training, validation and testing samples and make sure they have no overlaps (right panel of Figure A.2).
Wikipedia traffic The Wiki web page traffic records the daily number of visits for 550 Wikipedia pages from 2015-07-01 to 2016-12-31. The features are decoded from the able to obtain the project name, e.g. zh. zh.wikipedia.org. the access,
webpage url, where e.g. all-access, and the agent, c.g. spider. We use one-hot encoding to represent the features, and end up with a 14-dimension feature vector for each webpage.
The feature vectors not change with time, We use the feature vectors and traffic data from the past 200 days to predict the traffic of the next 14 days, which is also standard task for this dataset. The missing data are treated as zero.
Alibaba online shopping data. The dataset contains 4,136 online shopping sequences with a total of 1,552 items. Each shopping sequence has a varying number of time- stme-stamped user-item interactions, from 11 to 157. We consider the standard next-item recommendation task, where we make recommendation based on the past interactions. No user or item features are available.
Walmart.com e-commerce data The session-based online shopping data contains 122000 shopping sessions made by 1,000 frequent users, with a total of 2,034 items.
https://www.bcc-jiee www.gggeenaapg..eeeee mpp.de//wetter/
httpp::/www..t www.le..g......
24

---

Published as a conference paper at ICLR 2021
The lengths of the sessions vary from 14 to 87. In order to be consistent with the Alibaba online shopping data, we do not use the provided item and user features. We also consider the next-item recommendation task.
Preprocessing, Train-validation-te split and metric To ensure fair comparisons across the various models originated from different setting, we minimize the data-specific preprocessing steps, especially for the time series dataset.
Train
History
Predict
Train
Validation
Test
Validation
History
Predict
Test
History
Predict
Time
Time
Figure A.2: Left: the walk-forward split. Notice that we do not to specify the train-validation proportions when using the walk-forward split. Right: side-by-side split with moving window.
Jena weather dataset. We do the train setii--allldiin-tteee test split by 60%-20%-2000 on the time axis (right panel of Figure A.2). We first standardize the features the training data, and then use the mean and standard deviation computed on the training data to standardize the validation and testing data, so there is no information leak.
For Case 1, we use the observations made each hour (one every six observations) most recent 120 hours (5 days) to predict the temperature 12 hours in the future.
For Case 2, randomly sample 120 observations from the most recent 120 hours (with total of 720 observations), to predict the temperature 12 hours in the future.
For Case 3, we randomly sample 120 observations from the most recent 120 hours (with total of 720 observations), to predict the temperature randomly sampled from 12 hours (with total of 48 observations) in the future.
Wikipedia traffic. We use the walk-forward split (illustrated in the left panel of Figure A.2p test the performance of the proposed approaches under different wran-validationntee split schema. The relatively long compared with the full time series. For instance, on the Wiki traffic data, the full sequence length is 500, and the training sequence length is 200, so it is impossible to conduct the side-by-see bid split. The features are all one-ho hot encoding, so we do not carry out any preprocessing. The web traffics standardized in the same fashion the Jena weather data.
split is helpful when the length of training time series is
For Case 1, we use the most recent 200 observations to predict the full web traffics for 14 days in the future.
For Case 2, we randomly sample 200 observations from the past 500 days (with total of 500 observations), to predict the full web traffics for 14 days in the future.
For Case 3, we randomly sample 200 observations from the past 500 days (with total of 500 observations), to predict web traffics sampled from 0 to 14 days in the future.
Alibaba data. We conduct the standard preprocessing steps used in the recommender system literature. We first filter items that have less than 5 total occurrences, and then filter out shopping sequences that has less then 10 interactions. Using the standard train- validation-test split in sequential recommendation. for each shopping sequence, use all but the last two records as training data, the second-to-lane record as validation data, and the last record as testing data.
All the treated as the positive record. For each positive record (x<(1 k),..., (()) xe( 1)).
samples obtained from the real shopping sequences are
25

---

Published as a conference paper at ICLR 2021
randomly sample 100 items (= + 1)); and treat each (z(t ........ x(t), =z + 1)')
as a negative sample, which is again a standard implementation in the recommender systems where no negative labels are available.
Walmart.com data. We use the same preprocessing steps and train-validation-test split as the Alibaba data.
As for the metrics, we use the standard Mean absolute error (MAE) for the time-series prediction tasks. For the item recommendation tasks, we use the information retrieval metrics accuracy and discounted cumulative gain (DCG). Recall that for each shopping sequence, there is one positive sample and 100 negative samples. We rank the candidates {a(tt 1),a² (t + "0)...,Z(ooff according to the model's output score on each item. The accuracy checks whether the candidate with the highest score is the positive zr(1 1), and the DCG is given by 1/ log (rank(x(( + 1))) where rank (at(t + 1)) is the position at which the positive xxt + 1) is ranked.
1)}
B.2 MODEL CONFIGURATION AND IMPLEMENTATION DETAIL
We observe from the empirical results that using kernel addition (El)") (x,1;x', = [x) (x)xxx)xx) Kr-(x, x;'', (')) and kernel multiplication give similar results in our experiments, but kernel addition much faster in both training and inference. Therefore, choose use kernel addition as a trick to expedite the computation. Note that kernel addition corresponds to simply adding up the hidden representations from neural network, and the random features from the temporal kernel.
We first show the configuration and implementation for the models we use in time-series prediction. All the models take the same inputs for each experiment with the hyperparamters tuned on the validation data. Note that the VAR model do not have randomness in model initialization and training, so their outputs do not have standard deviations.
For the NN+time, NN+trigo and T-NN models, the temporal structures (time feature) are added to the same part of the neural architectures (illustrated in Figure I. and are made to have the same dimension 32 (except for NN+time). We will conduct sensitivity analysis later on the dimension.
For the proposed T-NN models, we treat the number of INN (affine coupling) blocks used in building the spectral distribution as tuning parameters. All the remaining model configurations are the same across all model variants. We do not experiment on using regularizations, dropouts or residual connections. In terms of reparameterization, we draw samples from the auxiliary distribution once at the beginning, so we do not have to resample during the training process.
VAR. In the vector autoregression model, each variable is modeled as a linear combination of past values of itself and the past values of other variables in the system. The order, i.e. the number of past values used, is treated as the hyperparameter, which we select according according to the AIC criteria the validation data.
For experiments on the Jena weather dataset, we choose the order from {20, 40, 60, 80, 100, 120}. since the maximum length of the history used for prediction is 120 for all three cases. Similarly, for experiments on the Wiki traffic dataset, we choose the order from (40, 60, ,o,.......000.
RNN models. We use the lay-r RNN model the standard RNN cells. The hidden dimension for the RNN models is selected to be 32 after tuning on the orignal model. To make time-series prediction, the output of the final state is then passed to a two-layer fully- connected multi-layer perceptron (MIP) with ReL.U as the activation function. We adjust the hidden dimensions of the MLP for each model variant such that they have approximately the same number of parameters.
CNN models. We adopt the CausalCNN (Wavenet) architecture from Oord al. (016). The original CausalCNN treats the number of filters, filter width, dilation rates and number of convolutional layers are hyperparameters. Here, to avoid the extensive parameter tuning for each model variant, we tune the above parameter of the plain CausalCNN model and adpot them to the other model variants. Specifically, we find that using 16 filters where each filter has width of 2, with 5 convolutional layers and the dilation rates given by {2 15 > gives the best result for all three cases. Similar to the RNN models, we then pass the output to two-layer MLP to obtain the prediction.
26

---

Published as a conference paper at ICLR 2021
Attention models. We use single attention block in the self- architecture Vaswani et al.] (2017). Unlike the ordinary attention in sequence-tus seeennce-tt-sepeee sequence learning. here we need to employ an extra causal mask (shown in Figure 1) to make sure that the model is not using the future information to predict the past. Other than that, we adopot the same key-query-valuee attention setting, with their dimension the tuning parameter. We find that dimension=1e 16 gives the best result in all cases for the original model and we adopt this setting the rest model variants. Also, pass the output to a two-layer MLP to obtain the prediction.
We now discuss the configurations and implementations for the recommendation models. We find out that the two-tower architecture illustrated in Figure is widely used for sequential recommendation (Hidasi et al. 2015 Li et al., 2017a). Each item is first passed through embedding layer, and the history is processed by some sequence processing model, while the target is transformed by some feed-forward neural network (FFN) such as the MLP. The outputs from the towers then combined together, often by concatenating, and then pass to the top-layer FFN to obtain a prediction score. Hence, adopt this complex neural architecture to examine our approaches, with the sequence processing model in Figure A.3] replaced by their time-aware counterparts given in Figure A.I
Prediction layers
FFN
concat| 0ea00n
Sequence learning layers
Sequence processing model
FFN
predict
Embedding layer
X,-k
Nt-k+: Xt-k+2
X
X
Figure A.3: A standard two-tower neural architecture for sequential recommendation. To incorporate the continuous-tie information, we adopt the model architectures in AnA. to replace the sequence processing model here.
To be consistent across the Alibaba and Walmart.com dataset, we use the same model configurations. Specifically, the dimension of the embedding layer is chosen to be 100, the FNN on the sequence learning layer a single-layer MLP with ReLU as activation, and the prediction layer FNN two- layer MLP with ReLU as activation. We keep the modules to be simple possible to avoid overcomplications in model selection. For the T-NN models, we treat the dimension of the random Fourier features Φ, and the number of INN (affine coupling) blocks used in building the spectral distribution as tuning parameters. We also do not experiment on using regularizations, dropouts or residual connections.
GRU-Rec. We use a single layer RNN with GRU cells the sequence processing model. We treat the hidden dimension of GRU as tuning parameter, and according to the validation performance evaluated by the accuracy, dimension=32 gives the best outcome for the plain GRUforRec, and we adopt it for all the RNN variants.
CNN-Rec. Similar the experiments for time-series prediction, we treat the number of convolutional layers, dilation rates, number of filters and filter width as tuning parameters. We select the best settings for the plain CNNforRec and adopt them to all the model variants.
We also use the single- single-block self-attention model as the sequence processing model. We treat the dimension of the key, query, value matrices (which are of the same dimension) as the turning parameter. turns out that dimension=20 gives the best validation performance for ATTNforRec, which adopt to all the model variants.
ATTN-Rec.
27

---

Published as a conference paper at ICLR 2021
phase training
phase inference
0.08
0.06
second
RFF-INN
0.04
time
2
0.02
0.00
CNN
RNN model
Attention
CNN
RNN model
Attention
Figure A.4: The training and inference speed comparisons for standard neural architectures (RNN, CausalCNN, self-attention) equipped with the proposed temporal kernel approach, and concatenating the time to feature vector.
Unlike the time-series data where the all the samples have a equal sequence length, the shopping sequences have various lengths. Therefore, we the maximum length to be 100 and use the masking layer to take account of the missing entries.
Finally, mention the implementation handle the different scales and formats of the timestamps. The scale of the timespan between events can be very different across datasets, and the absolute often less informative, they could be the linux epoch time or the
value of the timestamps calendar date. Hence, for each sequence given by { (I1,t1), (Ik. t&), (Ix+1, fx+1)} where the target is to predict (I*+1,(4+1) 1x+1), convert it to the representation under timespans: {((21, fx+1 t)))..... (I&a f tx), ( 0))- We then transform the scale of the timespans match with the problem setting, for instance, in the online shopping data the timespan is measured by the minute, and in the weather prediction it measured by the hour.
B.3 COMPUTATION
The VAR is implemented using the Python module statsmodel. The deep learning models implemented using Tensorflow 2.0 on a single Nvidia V100 GPU. We use the Adam optimizer and adopt the early-stopping training schema where the trainning is stopped if the validation metric stops improving for epochs. The loss function is mean absolute error for the time series prediction, and binary cross-entropy loss with the softmax function for the recommendation tasks. The final metrics are computed on the hold-out test data using model checkpoints saved during training that has the best validation performance.
We briefly discuss the computation efficiency of our approach. From Figure A.4 we see that the extra computation time for the proposed temporal kernel approach is almost negligible compared with concatenating time to the feature vector. The advantage is partly due to the fact that we draw samples from the auxiliary distribution only once during training. Note that this does not interfere with our theoretical results, and greatly speeds up the computation.
B.4 VISUAL RESULTS
We visualize the predictions of our approaches on the Jena weather data in Figure A.5 We see that T-RNN, T-CNN and T-ATTN all capture the temporal signal well. In general, T-RNN gives slightly better predictions on the Jena weather dataset.
28

---

Published as a conference paper at ICLR 2021
-15
pala
310
-2.
2000
$11.0
Figure A.5: The predictions of the T-RNN, T-CNN and T-ATTN approaches. We use the models trained for Case that predict the temperature hours the future. The plot is made by using sliding windows. The timestamp reflects the hours.
date Weather task case-]
Weather task case-2
data Weather task case-3
0.7
0.5
0.4
0.3
0.3
0.7
data Wiki task case-1
data Wili task - case-2
data Wiki task case-3
CNN
RNN
Attention
0.6
(189
0.4
0.3
0.2
0.1
RFF-IN
RFF-INN
Gousslen
AFF-INN
UFF-GGusslan
Figure A.6: The ablation study on parameterizing the spectral density with Gaussian distribution, or with INN.
B.5 EXTRA ABLATION STUDY
We show the effectiveness of using INN to characterize the temporal kernel, compared with using known distribution family. We focus on the Gaussian distribution for illustration, where both the mean and (diagonal of) the covariance are treated as free parameters. In short, we now parameterize the spectral distribution as Gaussian instead of using INN. We denote this approach by NN-RF, to differentiate with the temporal kernel approach T-NN.
First, we compare the results between T-NN and NN-RF on the time-series prediction tasks (shown in Tigure/A.6) We see that T-NN uniformly outperforms NN-RF. The results for the recommendation task is similar, as we show in Table A.3 where T-NN still achieves the best performance. The numerical results consistent with Theorem where more complex distribution family can lead to better outcome. also justifies our usage of the INN to parameterize the spectral distribution.
B.6 SENSITIVITY ANALYSIS
We conduct sensitivity analysis to reveal the stability of the proposed approaches with respect to the model selections and our proposed learning schema. Specifically, focus the dimension of the random features & and the number INN (affine coupling) blocks when parametrizing the spectral distribution. The results for the time-series prediction tasks are given in Figure A.7 A.8
29

---

Published as a conference paper at ICLR 2021
Table A.3: The ablation study on parameterizing the spectral density with Gaussian distribution, or with INN, when composing the recommendation models with temporal kernel. The reported are the aceuracy and discounted cumulative gain (DCG) for the nomain-specific models on temporal recommendation tasks.
Alibaba
Walmart
Metric
Accuracy
DCG
Accuracy
DCG
GRU-Rec-RF
78.051.22
47.30/.13
19.96/.15
4.82/.21
T-GRU-Ree
79.47/.3
49.82/.40
23.41/111
8,44/21
CNN-Réc-RU RF
75,231.35
44.607.26
16.337.20
2,74/19
T-CNN-Rec
76.45/.16
46.55/33
18,59/33
4.56/.31
ATTN-Rec-RF
52.30/.47
31.51//55
22.73/.41
7,95123
T-ATTN-Ree
53.49/.31
33.58/.30
25.511.17
9.22/.15
The sensitivity analysis results for the recommendation task Alibaba and Walmart.com datasets are provided in Figure A.9 A. From Table A.3 and A.2 we see that the ATTN-Rec models do not perform well on the Aliababa dataset, and the CNN-Rec models not perform well on the Walmart.com dataset. Therefore, we do not provide sensitivty analysis for those models on the corresponding dataset.
A.II
In general, the pattern is clear with respect to the dimension of o, that the larger the dimension (within the range we consider), the better the performance. The result is also reasonable, since larger dimension for é may express more temporal information and better characterize the temporal kernel On the other hand, the pattern for the number of INN (affine coupling layers) blocks less uniform, where in
cases too few INN blocks suffer from under-fitting, while in some other cases too may INN blocks lead to over-fitting. Therefore, we would recommend using the number of INN blocks hyperparameter, and keep the dimension of reasonably large.
30

---

Published as a conference paper at ICLR 2021
base model RNN
base model CNN
base model Attention
0.30
0.28
model
0.26
RNN RNN-trigo
T-CNN
0.24
CNN-trigo
T-Attention
0.22
Attention-trigo
0.20
16
32 dimension
128
128
16
32
64
128
dimeession
dimension
model
Attention
model
0.35
T-RNN
mac
RNN-triga
0.30
T-CNN
CNN-trigo
T-Attention
0.25
Attention-trigo
0.20
6
16
32 dimension
64
128
16
32 dimension
64
128
16
32
64
128
dimension
base model RNN
base model CNN
base model = Attention
0.54
0.52
T-RNN
RNN-trigo
0.50
CNN
0.48
T T-Attention
Attention-trip trigo
0.46
16
32
64
128
16
32 dimension
64
128
16
32 dimension
64
128
dimension
Figure A.7: Sensitivity analysis on the dimension of Φ., for the Jena weather dataset. From top to bottom are the results for Case 1, Case and Case 3 respectively.
31

---

Published as a conference paper at ICLR 2021
base model RNN
base model CNN
base model = Attention
model
0.46
mac
RNN-trigo
0.44
T CNN
CNN-trigo
T- T-Altention
Attention-trigo
0.40
32 dimension
dimension
dimensson
base model RNN
base model CNN
base model Attention
0.51
0.50
0.49
madel
T-RNN
0.48
RNN-trigo
0.47
T-CNN
CNN trigo
0.46
T-Attention
0.45
Attention -erioo
0.44
8
16
32
64
128
16
32 dimension
64
128
16
32 dimension
64
128
dimension
base model
base model CNN
base model Attention
0.54
T RNN
RNN-trigo
0.50
T-CNN
CNN-trigo
0.48
T-Attention
trigo
8
16
32
64
128
16
32 dimension
64
128
16
32 dimension
64
128
dimension
Figure A.8: Sensitivity analysis the dimension of Φ., for the Wikipedia web traffic dataset. From top to bottom are the results for Case 1, Case and Case 3 respectively.
task Alibaba
task = Walmart.com
28
82
26
80
accuracy
model
78
24
T-RNN-Rec
T-CNN-Rec
T-ATTN-Rec
20
4
#layers
# #layers
Figure A.9: Sensitivity analysis on the number INN (affine coupling) layers for the recommendation task on Aliaba and Walmart.com datasets.
32

---

Published as a conference paper at ICLR 2021
base model RNN
base model CNN
82
80
accuracy
model
T-RNN-Rec
78
RNN-Rec-trigo
T-CNN-Rec
76
CNN-Rec-trigo
74
32 dimension
128
32
64
128
dimension
base model RNN
base model CNN
28
accuracy
model
T-RNN-Rec
24
RNN-Rec-trigo
T-ATTN-Rec
22
ATTN-Rec-trigo
20
8
16
32
64
128
8
16
32 dimension
64
128
dimension
Figure A.10: Sensitivity analysis on the dimension of d, for the recommendation task on the Alibaba dataset upper panel and the Walmart.com dataset lower panel.
0.50
0.34
0.34
0.32
0.32
0.49
0.30
0.30
T-RNN
T-CNN
0.26
0.26
T-Attention
0.24
0.24
0.46
0.22
0.22
0.45
#layers
#layers
#layers
task case-1
task case-2
task case-3
0.48
0.52
0.54
0.52
0.45
0.50
model
T-RNN
0.44
0.48
0.43
T-Attention
0.46
0.42
0.46
0.41
0.44
4
2
#layers
Wlayers
#layers
Figure A.11: Sensitivity analysis on the number of INN (affine coupling) layers. The upper panel gives the results on the Jena weather dataset, the lower panel gives the results on the Wikipedia web traffic dataset.
33


Reviews:
[{"review_id": "ZKTMBwzvLav", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##### Post-rebuttal update\n\nI've read the rebuttal and updated my score.\n\n---------------\n\nThis paper proposes a deep learning model for incorporating temporal information by composing the NN-GP kernel and a temporal stationary kernel through a product. The temporal stationary kernel is represented using its spectral density, which is parameterized by an invertible neural network model. This kernel will be learned from the training data to characterize its temporal dynamics.\n\n##### Originality & Significance\nThe modeling approach taken in this paper is original to my best knowledge. Although it is well-known that second-order stationary processes has a SDE correspondence, it is rare that this property is connected to NN as GPs and this work finds an application where such ideas can be potentially useful. However, I find it difficult to say anything about significance of this idea since it is not very clearly described. I encourage the authors to make a substantial revision to clarify the issues listed below.\n\n##### Clarity\nThe clarity is low. Although the motivation and the high-level idea is clear, I find it very difficult to understand the actual approach taken by this work. There is no description of the actual algorithm and I can see many algorithmic and computational issues left without discussion:\n* How is prediction made at a specific (t, x)? Do you use a GP predictive mean conditioned on the training points?\n* If the prediction is made by GPs, how do you solve the scalability issue? When the training set is large, do you take a sparse approach? The temporal kernel is defined through a random feature representation, do you take advantage of it for fast computation?\n* or you just take a weight-space approach and compose the features (take pairwise product of the features of k_T and \\Sigma to form the new features)?\n* Is NN-GP or NTK kernels used to compute the kernels? How do you compute them? Do you use a Monte-Carlo estimate or the closed-form (computed through a recursion)? \nI will be happy to raise the score if these questions are properly addressed.\n\n##### Strengths\n* The modeling approach is novel.\n* The proposed method consistently outperforms other baselines in handling irregular continuous-time data.\n\n##### Weaknesses\n* The method used is not clearly described.\n* The non-stationary extension to Bochner's theorem is a known result.\n* Although the performance is shown to outperform other NN-based approaches in experiments, there might be scalability issues to apply the approach to larger-scale problems with long sequences (assume a non-weight-space approach).\n\n##### Minor\nP16: A.4.1: \"We the Fourier transformation transformation on both sides\"?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LksBrZh5pD1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This article proposes a methodology to *adapt* NNs to continuous-time data though the use of a (temporal) reproducing kernel. I enjoyed reading the paper, the message is clear, illustrative and the connection with other existing works is to the point. Although I am unfortunately unable to assess the theoretical novelty of the paper (I am unaware of the details of the state of the art in the subject) the contribution of the paper relates to the study of a kernel, given by an ODE, attached to the input of the NN. This kernel is also represented using Fourier feature expansions. \n\nThough the paper heavily relies on well-known concepts  (standard NN, GPs, Fourier features), I see that is has a contribution. \n\nI suggest the following amendments:\n-for some readers, the general proposed architecture might be confusing. Perhaps a diagrams (similar to that in Fig A.1) would be useful in the first pages of the paper. How does the kernel turn the continuous-time data into NN-ready?\n-much useful material is relegated to the appendix, if key results, scope and more are only in the appendix, they might not receive the deserved attention.\n-please better clarify how different your work is from the existing literature: NTK, deep kernel learning, neural ODEs, etc\n  \n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "IC8nQwTunHp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a general deep learning method with temporal-kernel for continuous time series modeling.\n\nThe proposed method is technically sound and solid. The decomposition of the neural and temporal kernel brings together the kernel methods and deep learning, which delivers a general and fundamental solution to time series, especially the irregularly sampled ones or those with missing values.\nIn brief, this work may demonstrate a promising way of handling such problems and inspires and encourages other research in this direction.\n\nThe writing is thorough and clear. Though I did not check all proofs and the supplementary, the descriptions and arguments in the paper are properly delivered.\n\nThe proposed model consistently outperforms RNN, TCN, and attention baselines on a variety of datasets. The settings of the case 2/3 are reasonable.\nBesides, it is interesting to see that the speed is still comparable to the baselines.\n\nHowever, in my opinion, more baseline comparisons need to be added. I did not quite buy the claim that the proposed method is not compared with recurrent neural ODE-type models and point process models because of its more generalization and flexibility.\n\nMinor typos:\nIn page 3: infitnitesimal -> infinitesimal; covaraince -> covariance\nThe font size in figures may be increased for better readability.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9YipEv-204i", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The ms introduces a time component in the traditional NN setup, where the hidden layers change dynamically according to time. The idea is to borrow strength from the newly introduced time dimension to improve prediction performance.\n\nOne of the key idea is to treat each hidden layer in NN as a Gaussian process, which is represented  as “neural network kernel”. Functions drawn from this Gaussian process at different time points are assumed to follow a continuous-time system, which is actually a ODE. For some reason it is difficult to use the continuous-time system to compute the temporal kernel directly in the time domain. The ms proposes to convert the functions to the frequency domain, which leads to a nice property such that we can compute a temporal kernel (Eq. (2)) in frequency/spectral domain. Hidden layer of NN at different time points can be seen as a large Gaussian process, whose kernel could be composed by the aforementioned NN kernel and temporal kernel (Eq. (5)). This decoupling is the key of this ms. \n\nRegarding the computation of the kernels, NN kernel can be computed by extracting the features of the hidden layer. The temporal kernel can be computed by sampling the spectral distribution, which is called the random feature representation (Eq. (7)). However, it is not clear how to specify the spectral distribution. In all examples, Normal distribution is used, which is OK but may not be able to capture the complexity of ODE. \n\nThe ms applies the proposed method to real dataset and achieve better performance than baseline methods in prediction tasks involving irregular time points setup.\n\nIn general I feel this is a nice paper. The idea of learning NN and time dynamics at the same time seems to be useful in many applications. The ms cleverly decouples the learning of NN kernel and temporal kernel in two independent modules, which can maximumly utilise current implementations. \n\nHowever, due to my limited knowledge in signal processing, I am not able to dig into the mathematical details and make strong recommendations (especially Claim 2). \n\nSome minor comments\n1. What is the purpose of Claim 1? From the supplementary it just shows that f(t) and f[i] are not equal, but they may be very close and does not have a huge impact of the result. There are lots of approximations in other parts of the model.\n2. Why f(t) needs to be ODE ?\n3. Page 3, section 3, line 2, in the formula of f(iw), the second derivation term seems to be missing\n4. Page 4, 5 lines after Eq. (3), a_2(x)!=0 => a_1(x)!=0\n5. Page 5, Eq. 7, cos(tw_n) => cos(tw_m)\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel approach for integrating time into deep neural network models based on the Gaussian process limit view of a neural network model. Specifically, the approach augments an a-temporal neural network designed to process a single time point with a temporal kernel that relates data points across time. The composition of the a-temporal neural network kernel with with the temporal kernel is accomplished efficiently using a random features representation of the temporal kernel. The authors propose to represent the temporal kernel via its spectral decomposition, which makes the approach quite flexible. Learning leverages re-parameterization. While random features have been used to approximate temporal kernels in prior work [1], the approach in this paper is significantly more general in that it can be composed with any a-temporal deep architecture and the authors show results for RNNs, CNNs, and attention-based models. The predictive performance of the approach also appears to be consistently better than baselines and it works particularly well on the challenging case of irregularly sampled data.

In terms of weaknesses, the reviewers had a number of questions about the paper. The authors updated the paper to include some more recent models including ODE-RNNs. This material is currently presented in the appendices and needs to be moved into the main paper. Several of the reviewers also had technical questions questions that are in fact addressed in the manuscript; however, the authors are relying heavily on the appendices to present many important details and the paper is currently over 30 pages long. The frequent references to the appendix for additional details makes the paper a challenging read. The authors have already done some work to address clarity by adding a new figure, but should prioritize moving additional key details into the main body of the paper to improve readability. 

[1] http://auai.org/uai2015/proceedings/papers/41.pdf
Decision: Accept (Poster)