Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##### Post-rebuttal update

I've read the rebuttal and updated my score.

---------------

This paper proposes a deep learning model for incorporating temporal information by composing the NN-GP kernel and a temporal stationary kernel through a product. The temporal stationary kernel is represented using its spectral density, which is parameterized by an invertible neural network model. This kernel will be learned from the training data to characterize its temporal dynamics.

##### Originality & Significance
The modeling approach taken in this paper is original to my best knowledge. Although it is well-known that second-order stationary processes has a SDE correspondence, it is rare that this property is connected to NN as GPs and this work finds an application where such ideas can be potentially useful. However, I find it difficult to say anything about significance of this idea since it is not very clearly described. I encourage the authors to make a substantial revision to clarify the issues listed below.

##### Clarity
The clarity is low. Although the motivation and the high-level idea is clear, I find it very difficult to understand the actual approach taken by this work. There is no description of the actual algorithm and I can see many algorithmic and computational issues left without discussion:
* How is prediction made at a specific (t, x)? Do you use a GP predictive mean conditioned on the training points?
* If the prediction is made by GPs, how do you solve the scalability issue? When the training set is large, do you take a sparse approach? The temporal kernel is defined through a random feature representation, do you take advantage of it for fast computation?
* or you just take a weight-space approach and compose the features (take pairwise product of the features of k_T and \Sigma to form the new features)?
* Is NN-GP or NTK kernels used to compute the kernels? How do you compute them? Do you use a Monte-Carlo estimate or the closed-form (computed through a recursion)? 
I will be happy to raise the score if these questions are properly addressed.

##### Strengths
* The modeling approach is novel.
* The proposed method consistently outperforms other baselines in handling irregular continuous-time data.

##### Weaknesses
* The method used is not clearly described.
* The non-stationary extension to Bochner's theorem is a known result.
* Although the performance is shown to outperform other NN-based approaches in experiments, there might be scalability issues to apply the approach to larger-scale problems with long sequences (assume a non-weight-space approach).

##### Minor
P16: A.4.1: "We the Fourier transformation transformation on both sides"?

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This article proposes a methodology to *adapt* NNs to continuous-time data though the use of a (temporal) reproducing kernel. I enjoyed reading the paper, the message is clear, illustrative and the connection with other existing works is to the point. Although I am unfortunately unable to assess the theoretical novelty of the paper (I am unaware of the details of the state of the art in the subject) the contribution of the paper relates to the study of a kernel, given by an ODE, attached to the input of the NN. This kernel is also represented using Fourier feature expansions. 

Though the paper heavily relies on well-known concepts  (standard NN, GPs, Fourier features), I see that is has a contribution. 

I suggest the following amendments:
-for some readers, the general proposed architecture might be confusing. Perhaps a diagrams (similar to that in Fig A.1) would be useful in the first pages of the paper. How does the kernel turn the continuous-time data into NN-ready?
-much useful material is relegated to the appendix, if key results, scope and more are only in the appendix, they might not receive the deserved attention.
-please better clarify how different your work is from the existing literature: NTK, deep kernel learning, neural ODEs, etc
  
 
review_title: Well written, connected with the current literature, appropriate experimental validation

final_score: rating: 7: Good paper, accept
review_content: The ms introduces a time component in the traditional NN setup, where the hidden layers change dynamically according to time. The idea is to borrow strength from the newly introduced time dimension to improve prediction performance.

One of the key idea is to treat each hidden layer in NN as a Gaussian process, which is represented  as “neural network kernel”. Functions drawn from this Gaussian process at different time points are assumed to follow a continuous-time system, which is actually a ODE. For some reason it is difficult to use the continuous-time system to compute the temporal kernel directly in the time domain. The ms proposes to convert the functions to the frequency domain, which leads to a nice property such that we can compute a temporal kernel (Eq. (2)) in frequency/spectral domain. Hidden layer of NN at different time points can be seen as a large Gaussian process, whose kernel could be composed by the aforementioned NN kernel and temporal kernel (Eq. (5)). This decoupling is the key of this ms. 

Regarding the computation of the kernels, NN kernel can be computed by extracting the features of the hidden layer. The temporal kernel can be computed by sampling the spectral distribution, which is called the random feature representation (Eq. (7)). However, it is not clear how to specify the spectral distribution. In all examples, Normal distribution is used, which is OK but may not be able to capture the complexity of ODE. 

The ms applies the proposed method to real dataset and achieve better performance than baseline methods in prediction tasks involving irregular time points setup.

In general I feel this is a nice paper. The idea of learning NN and time dynamics at the same time seems to be useful in many applications. The ms cleverly decouples the learning of NN kernel and temporal kernel in two independent modules, which can maximumly utilise current implementations. 

However, due to my limited knowledge in signal processing, I am not able to dig into the mathematical details and make strong recommendations (especially Claim 2). 

Some minor comments
1. What is the purpose of Claim 1? From the supplementary it just shows that f(t) and f[i] are not equal, but they may be very close and does not have a huge impact of the result. There are lots of approximations in other parts of the model.
2. Why f(t) needs to be ODE ?
3. Page 3, section 3, line 2, in the formula of f(iw), the second derivation term seems to be missing
4. Page 4, 5 lines after Eq. (3), a_2(x)!=0 => a_1(x)!=0
5. Page 5, Eq. 7, cos(tw_n) => cos(tw_m)

review_title: Clever treatment to decouple NN and temporal dynamics

final_score: rating: 7: Good paper, accept
review_content: This paper proposed a general deep learning method with temporal-kernel for continuous time series modeling.

The proposed method is technically sound and solid. The decomposition of the neural and temporal kernel brings together the kernel methods and deep learning, which delivers a general and fundamental solution to time series, especially the irregularly sampled ones or those with missing values.
In brief, this work may demonstrate a promising way of handling such problems and inspires and encourages other research in this direction.

The writing is thorough and clear. Though I did not check all proofs and the supplementary, the descriptions and arguments in the paper are properly delivered.

The proposed model consistently outperforms RNN, TCN, and attention baselines on a variety of datasets. The settings of the case 2/3 are reasonable.
Besides, it is interesting to see that the speed is still comparable to the baselines.

However, in my opinion, more baseline comparisons need to be added. I did not quite buy the claim that the proposed method is not compared with recurrent neural ODE-type models and point process models because of its more generalization and flexibility.

Minor typos:
In page 3: infitnitesimal -> infinitesimal; covaraince -> covariance
The font size in figures may be increased for better readability.

review_title: Review from R3

Meta-review: This paper presents a novel approach for integrating time into deep neural network models based on the Gaussian process limit view of a neural network model. Specifically, the approach augments an a-temporal neural network designed to process a single time point with a temporal kernel that relates data points across time. The composition of the a-temporal neural network kernel with with the temporal kernel is accomplished efficiently using a random features representation of the temporal kernel. The authors propose to represent the temporal kernel via its spectral decomposition, which makes the approach quite flexible. Learning leverages re-parameterization. While random features have been used to approximate temporal kernels in prior work [1], the approach in this paper is significantly more general in that it can be composed with any a-temporal deep architecture and the authors show results for RNNs, CNNs, and attention-based models. The predictive performance of the approach also appears to be consistently better than baselines and it works particularly well on the challenging case of irregularly sampled data.

In terms of weaknesses, the reviewers had a number of questions about the paper. The authors updated the paper to include some more recent models including ODE-RNNs. This material is currently presented in the appendices and needs to be moved into the main paper. Several of the reviewers also had technical questions questions that are in fact addressed in the manuscript; however, the authors are relying heavily on the appendices to present many important details and the paper is currently over 30 pages long. The frequent references to the appendix for additional details makes the paper a challenging read. The authors have already done some work to address clarity by adding a new figure, but should prioritize moving additional key details into the main body of the paper to improve readability. 

[1] http://auai.org/uai2015/proceedings/papers/41.pdf
Decision: Accept (Poster)