Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces convolutions with continuously parametrized kernels for sequential data. Continuous kernel convolutions, CKConv for short, parametrize the kernel associated to a convolutional layer as a continuous mapping, $\psi: \mathbb{R}^+ \rightarrow \mathbb{R}^{N_{out} \times N_{in}}$, from the relative time offset $\Delta\tau \in \mathbb{R}^+$ to a weights matrix $\mathbf{W} \in \mathbb{R}^{N_{out} \times N_{in}}$, i.e. $\psi(\Delta \tau) = \mathbf{W}$. This is in contrast to traditional CNN approaches, which represent the convolution kernel as an explicitly learnable set of weights over an a-priori fixed receptive field.

The corresponding CKConv operation can be applied
1) over arbitrarily long receptive fields, including a global one over the history of the input sequence, without the heavy memory burden incurred by representing the convolution kernel explicitly as a set of weights along the horizon;
2) to sequences that are sampled irregularly or asynchronously, since the admissible set of relative positions are not fixed in advance.

A relation is given, which shows approximately how the CKConv changes under resampling the input sequence (the output gets rescaled by the ratio of the sampling rates), which is suggested by the authors to support well-behavedness under varying input resolutions during training and/or between training and testing. Afterwards, a discussion is given about the parametrization of the kernel. Specifically, various MLP models are considered with different activation functions, such as, ReLU, LeakyReLU and Swish, and it is demonstrated that these MLPs are unable to learn functions with high-frequency oscillations, while a recently introduced Sine activation is able to. 

The experimental evaluation includes 1) classic stress tests for RNNs, 2) discrete sequence tasks (sMNIST, pMNIST, sCIFAR10), 3) time series modelling (CharacterTrajectories, Speech Commands) , where additionally robustness to missing/irregularly-sampled data and resolution changes are also investigated. The proposed model overall performs favourably to common alternatives; and it seems robust with respect to missing data, and moderate resolution changes.
main_review: # Review
## Clarity and novelty 
This is overall a well-written paper, which clearly discusses the limitations of popular sequence models such as CNNs and RNNs, and explains how the proposed CKConv model is able to provide improvements in those aspects. The related work seems to be adequately cited. The main idea, which is to represent the convolution kernel implicitly via parametrizing it by a neural network, is not novel in the literature, but it has not been applied previously for sequential data modelling as far as I know. 
***
## Strengths
The main value of the paper for me is in:
1) adapting the continuous kernel convolution model to the sequential data context
   + the idea of using a CKConv with a global memory horizon seems particularly useful, remedying the problem of not being able to learn long-range relationships, typically associated with RNNs and fixed horizon (shallow) CNNs
   + the ability to generate kernel weights for arbitrary relative offsets naturally also lends itself to learning from irregularly sampled data, which is non-trivial to address with classic approaches
2) the effort put into the empirical evaluation
   + observing that convolution kernels might naturally contain high-frequency oscillations, which are difficult to learn via common activations, and identifying MLPs with Sine activations as a suitable candidate for representing the kernel
   + carrying out the empirical evaluation across a range of tasks, which show strong performance of the proposed model
***
## Drawbacks
My criticism is regarding some of the claims and conclusions drawn. See these points below.

**Insensitivity to resolution changes**
It seems that the relation in equation (5) does not exactly support that the model is robust with respect to larger resolution changes. This is also shown in Table 5, where the performance degrades significantly if there is an order of magnitude difference between training and testing resolutions. This is of course because the discretized convolution in eq.(2) is not a proper discretization of the integral in eq.(1). In particular, if we consider a high-frequency limit (that is, sampling a continuous signal at increasing frequencies), then the summation in (2) does not converge. I wonder why the authors chose to omit the normalization from eq.(2)? They could have normalized by the number samples (or equivalently, by the sampling rate) for equispaced samples, or for irregular data simply multiply the summand by the local step size to obtain Riemann sums that are actually convergent, and hence mapping that is continuous-time convergent and invariant to resolution changes, which it is not right now.

**Asynchronous sampling**
It seems to me that regarding asynchronously sampled data, the model formulation is equivalent to combining zero imputation for the missing values with concatenating an observation mask to the input as extra channels. This is a valid approach, but not specific to the model at hand.

**Activations in $\texttt{MLP}^\psi$**
I found the discussion regarding the activation functions in the kernel parametrization quite intriguing. In particular, the experiments shown in Figure 4 coincide with the findings of "On the Spectral Bias of Neural Networks", Rahaman et al. 2019, where it is observed that NNs with ReLU activations trained under GD are biased towards learning low-frequency functions. It is also an interesting finding that the Sine activation seems to be able to overcome this limitation in this setting. But then one of the main points the paper tries to communicate seems to be that _Neural networks parameterizing spatial functions should use Sine nonlinearities_, which in my opinion paints too dark of a picture about ReLU and its companions. Especially so, as there have been approaches proposed in the literature to circumvent this limitation without changing the activation itself, see "Fourier Features Let Networks Learn
High Frequency Functions in Low Dimensional Domains", Tancik et al. 2020. Without properly testing out if these techniques do help the usual activations, the aforementioned conclusion stands on shaky grounds. Especially so, since ReLUs are well-studied in the literature, while Sine nonlinearities in NNs (i.e. SIRENs) seem to be somewhat of a wildcard. In particular, by the discussion in App. E.3 I am left wondering if SIRENs might be, on the other hand, prone towards learning (unnecessarily) high-frequency functions? 

Concretely, I would be curious to know how the conclusions of the experiment in Figure 4, and the ablation study in App. D.1 change if the relative time offset $\Delta \tau$ is first passed through a positional encoding or Random Fourier Feature mapping as done in the previously referenced paper. Furthermore, in case any of these preprocessings do help the "standard" activations, it would be also good to know if the learned functions still contain the high-frequency oscillations beyond the input time grid as mentioned in App. E.3, or if these end up being smoother?

**Modelling long-range relationships**
The authors have noted that dilated convolutions are able to model long-range interactions. I wonder why it wasn't included among the experiments? Clearly, to achieve a larger receptive field, dilated convolutions stack many layers, which means that the inputs are passed through several nonlinear activations before reaching the final representation the prediction layer receives. On the other hand, the CKConvs can increase the receptive field while keeping the model shallow. The benefits of this are not really clear at the moment, and whether the way the sequential information is processed in the dilated model (by passing it through "extra depth") has adversarial effects for the quality of the results.
Additionally, self-attention models are also able to capture long-range dependencies in their input sequences, so it would be worthwhile to include a discussion about them, and compare against it on more experiments.
summary_of_the_review: I am in favour of the paper mainly due to the compelling empirical performance, which should be of interest to the community, although some additional baselines and experiments could have been included.  It is clearly written with figures and illustrations to aid the reader. The main idea is not novel, but it has not been adapted to handling sequential data before, and it seemingly has many benefits in this context.  My criticism is regarding some discussions in the paper, and that some properties of the model seem to be overstated.

POST-REBUTTAL: Given the comprehensive answers given by the authors, and the implemented changes in the paper, I am inclined to raise my rating to an 8 from the previous 6.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This contribution adapts to sequential data the paradigm of continuous
convolution kernels in convolutional neural networks (CNNs). While
usual kernels are an extensive list of weights (one for each
position), the introduced CKConv frames these kernels as continuous 1D
functions, parameterized by a small multilayer perceptron. This
paradigms allows for wider kernels with fewer parameters, making it
possible to deal with long-range dependencies without
recurrence. Several experiments are provided and show that the
resulting CNN achieves state of the art performances on a variety of
tasks, and in particular is able to deal with irregularly sampled
data.

main_review: The contribution is clear and well written. I reviewed a previous
version for another venue and thank the authors for their effort in
clarifying their manuscript, which I now find very didactic and
pleasant to read.

I found the concept of continuous kernel interesting. The authors make
it clear that they are adapting it from previous work on 3D data, but
also show how its transfer to sequential data is a useful contribution
in particular for dealing with long-range dependencies without the
limitations of RNN or discrete CNN. In particular, they clearly show
how a discrete convolution kernel would require much more parameters
than a CKConv to model the same range of dependencies.

They also provide thorough experiments illustrating the superiority of
their proposed continuous kernel CNN (CKCNN) against state of the art
approaches on all targeted applications: long-range dependency
modelling, irregular sampling, varying sampling rate.

I believed that this contribution could both be inspiring for new
paradigms of networks (e.g. as an alternative or complement to
attention), and lead to better performances in practice on the
targeted applications.

One point that could be further clarified is the choice of using a single MLP
for modelling the Nin x Nout convolution kernels. Among other
possibilities, a separate network could be used for each of the Nout
convolution filters (i.e., one MLP instead of one discrete Nin x
kernel_size filter). Using a single MLP likely creates dependencies
among the Nout filters, is there a justification for this choice?

summary_of_the_review: The contribution is original and could have a large impact both because it leads to better performances in several important situations (including long range dependencies) and because it could inspire new designs. The proposed method is thoroughly evaluated.
The submission is also clear and well written.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes continuous convolutional kernels parametrized in the form of an MLP. The MLP gets the relative time as input and outputs the column of the convolutional kernel at the given relative time. The authors show that Sine-based (SIREN) non-linearity is best suited for the kernel generating MLP, and an experimental evaluation is performed.
main_review: **Pros:** 
- Interesting concept of representing arbitrary-sized convolutional kernels.
- The paper is well written, and the method's limitations are outlined clearly
- Great evaluation of how a SIREN-based parametrization is best suited for realizing the MLP. 

**Cons:**
- No code is provided during the review (although the authors claim to release it)
- The experimental evaluation mostly consists of equidistantly sampled fixed-length time series (sMNIST, etc.), i.e., data for which CKConv provides only little benefits over standard CNNs.
- Experimental evaluation with attention-based architectures on irregularly sampled variable-length sequences datasets, e.g., PhysioNet, MIMIC-III, and the Human Activity dataset, from Shukla et al. 2021 would have demonstrated the advantages and limitations much better.

**Questions**:
- How was the sequence reduced to a single prediction in the sequence classification tasks (sMNIST, etc.)?
- The paper Gu et al. 2020 "HiPPO: Recurrent Memory with Optimal Polynomial Projections" was published at NeurIPS. Please update the reference.

Shukla et al. Multi-Time Attention Networks for Irregularly Sampled Time Series. ICLR 2021
summary_of_the_review: I am in favor of acceptance as the concept of CKConv provides enough contribution, although the experimental evaluation would have been stronger with standard irregularly time-series datasets (PhysioNet, MIMIC-III)
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers convolutional networks for sequence processing and suggests to parameterize convolutional kernels in this setting by small neural networks. These neural networks are similar to implicit networks in the sense that its input represents a time difference to the currently evaluated time point in a sequence. For each time difference, the associated weights of the convolutional kernel are computed.

This allows to handle arbitrarily large convolutional kernels (albeit the desired size needs to be determined somewhat deterministically apriori), to handle irregularly sampled data and to handle data at different resolutions. Good results on a set of experiments show the practical validity of the idea. The paper does a larger study into the importance of the type of activation functions. 


main_review: This is a nice paper with one clear idea, which is demonstrated to work well. I like your detailed look at activation functions -- I think you should have a look at "Gegenbauer Polynomials". A potential weakness of the paper is it's limited set of experiments: Compared to the 'baseline' paper Bai 2018a, several datasets are missing, and, given the progression of time, several larger-scale datasets e.g. from speech or the financial domain could be added. 
summary_of_the_review: The described idea is a nice transfer from existing work of implicit neural network representations and continuous kernel formulation. Showing the importance of sine activation functions is very helpful to direct attention to the role of activation functions, depending on the actual modeling task. The empirical validation should be larger, though.
review_title: null

Meta-review: This paper introduces a convolution where the kernel is parametrised continuously over time (in the context of recurrent networks) to address vanishing gradients issues, by using another neural network to generate the kernels.
This is a meaningful idea, addressing an important problem.
The paper is well written and clear. The idea is novel (parametrised kernel already exist, but the way it's used here is new).
The experimental section is solid, although some reviewers suggests it could be extended with more baselines.
All reviewers recommend to accept the paper, therefore I also recommend accept.
Decision: Accept (Poster)