Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The article proposes a methodology for designing covariances kernels in a nonparametric way using Volterra series. The ideas proposed build on existing works on convolution models, Volterra kernels, SysID and MOGP. The paper substantially differ from previous works and the contribution is validated empirically. 
main_review: The authors deliver a thorough and clear literature review, which makes it easy for the reader to identify the motivation and the contribution of this article. As other works in the last decade, this paper focuses on kernel design for GPs, in particular, this kernel takes a nonparametric perspective, a practice that has done in the past but only by few researchers. The main distinction of this work is (in addition to the nonarametirc kernel construction) its application for multiple channels and the inclusion of nonlinearities

In my opinion, this paper is appropriate for NeurIPS but I would appreciate if the authors referred to the following points 

- The authors use a fixed grid of the input and kernel, please elaborate why this does not constitute a discrete/parametric model. 

- The authors refer several times to the similarities between the proposed method and GPCM. Is the approximate inference the same for both? (structured VI) or the implementation of GPCM in this paper is "enhanced" wrt to the original one? 

- The approaches for nonparametric kernels are known by their non-trivial implementation, perhaps the authors would make their code available so that the public can use their method. 

- Please elaborate on the computational complexity of the method, hopefully quantitatively. Due to the involved high-dimensional elements, this might be worth considering.

- The proposed method is designed for non-Gaussian data. Then, in the experiments, it seems that the authors implicitly assume that real-world data is non-Gaussian (it's most likely the case). Why not explain which nonlinearities are modelled by the Volterra approach?  



minor: 
l77: Voterra -> Volterra

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to model volterra kernels with Gaussian processes and develops an inference algorithm based on variational inference. 
main_review: 

1. square integrability of greens functions (line 66).

What is required of G to ensure finite output depends on G. That is, if u is bounded then it is sufficient that G is integrable (BIBO stability). 

2. Decaying square exponential (line 100-101).

Square integrability of G is sufficient for Tobar [29] because it is convolved with a white noise signal. In the present situation it is not clear that this is works. For example, take the second term in the sum of eq (2):  

int  ( int G_{d,2}(t-s_1,t-s_2) u(s_1) d s_1` )  u(s_2) d s_2 

which can be made finite if  u is in L_2 and  int G_{d,2}(t-s_1,t-s_2) u(s_1) d s_1` )   is an L_2 function for every t, but is this ensured by square integrability of G? 

3. Typo (Line 157). 

Lieber -> Leibler.


4. Number of inducing points (Line 240).

The number of inducing points assigned to the filters appears counterintuitive. You need few points to cover a small space and many points to cover a large space but the number of points assigned to the filters is directly opposite to this?

5. Length scale and noise parameters (line 249). 

It is not made clear how the length scales are fixed or how the noise hyperparameters are post-hoc estimated. 





limitations_and_societal_impact: yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper places GP priors over the kernels of a Volterra series which is driven by an input process drawn from a GP, which may optionally have associated observations (the IO variant of the model). Sampling and mean field variational inference procedures are described. The method is demonstrated on two real data sets.
main_review: The paper seemed to be inspired by [2]. However, it moved away from the full GP approximation to directly model the the output function via equation (5). This make the paper more original and significant. Sampling and inference procedure are clearly described. 

# COMMENTS
1. It will be worthwhile to see if (Markov chain) Monte Carlo posterior sampling of the model can yield better results than currently reported. This will allow the reader to assess if the limitation is because of the model or because of the approximate inference. Section 3.1 is a good start to build this upon. In the least, a comment on this aspect will be useful for the reader.

2. In the experiments, a number of runs via different random seeds is needed, and the best results are reported (though see point 3).  Does the various runs give drastically different results? What is the main resultant source of randomness: the gradient descent; the Monte Carlo estimate of eq (9); or the sampling procedure in eq (5)?

3. Section 5.3 says the best results from five runs are picked, but Table 3 shows standard deviation across three runs. Please clarify or correct.

4. Figure 5 is not very useful without the actual data also plotted in the test region; and it is not referred to at all the the text. Suggest to remove this, and use the space to expand on how one may interpret the kernels learnt in section 5.2.

5. Another future work could be to move beyond mean field approximation.

Changes after rebuttal
=================
4. Figure 5 does indeed have the actual data plotted. It was not very clear, and I missed it in the first reading. Suggest the authors to make this clearer in the revised version.
limitations_and_societal_impact: # Societal Impacts
Suggest to include some possible impact on the modelling aspect. For example, climate/weather modelling which is actually one of the examples.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper expands previous work to learn convolutional operators that map a latent (unobserved) GP to observed data. The convolutional operators/kernels are more general as in previous works (non-factorizing and non-homogeneous kernels), and inference in made scalable via a combination of a variation approach and recent results to efficiently sample GPs. Furthermore, the authors provide an extension, where the latent input process is also associated with data.
main_review: I found the paper interesting and very well written. It is providing some methodological expansion on previous work, such that less assumption on the Gaussian process convolutional models (GPCM) are needed. While the expansions of the model are straightforward, the variational inference of the model needed to utilize some recent findings on GP-sampling to get an efficient approximation of the ELBO. To use the sampling technique from [Wilson et al, 2020] the authors transform a GP with non-stationary kernels, to a stationary one. Then, the nontractable parts of the ELBO can be computed efficiently, and standard stochastic VI methods can be employed.
The experiments seem to be done thoroughly. However, I’d have some suggestion for improvements (see below). 
Overall, the paper is not the most innovative one, since novelty mainly in combinations of recent findings and an interesting, but straightforward model extension. But ideas are executed neatly, and I consider it as some solid work, which could be published at this stage.

## Update

I thank the authors for their response. Considering the additional information and experiments, that will be added to the final publication, I am happy to increase my score to 7.
limitations_and_societal_impact: + l. 153: I don’t think, one can say that the optimal variational distributions are Gaussians. Optimal would be, that they turn out to be Gaussians under the factorizing assumption of Eq. (7). Here, however, I believe it is an assumption that needs to be made.
+ Sec.5.1: While I agree, that the synthetic example is a difficult one, it would be insightful, if an inference example can be shown, that is coming from the generative model. Can the model then correctly identify C? Or does the fact, that the authors choose a decreasing number of inducing points for larger C’s corrupt the model selection?
+ Also for the real experiments, it would be nice to choose an example, where learnt filters can be interpreted in a meaningful way, and potentially demonstrate, why it is important to assume non-homogenous and/or non-factorizing filters, instead of just referring to the likelihood.
+ As discussed, I also consider the fact, that the input process u needs to be univariate the strongest restriction. I would appreciate, in the discussion, how many dimensions are likely to be feasible.
+ I am missing the report of runtimes in the experiments. How does it scale with number of data points, C, etc. How does it compare to the more limited models? Does one pay a high price in terms of computation time for the proposed model?
review_title: null

Meta-review: The reviewers found the paper interesting and well written, with a thorough and clear literature review, and methodologically sound. All reviewers recommend acceptance, and the remaining concerns are possible to address in the final camera-ready version. I recommend that the authors carefully go through the manuscript in light of the reviewer comments.

Decision: Accept (Poster)