Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: # Summary

To work with data that is not sampled on a grid, this work represents the activations of a neural network using a Gaussian process with RBF covariance. Convolutions can be applied in closed form, but nonlinearities get projected to another RBF-posterior GP by

1) calculating the first two moments of the post-nonlinearity activation, and
2) projecting the moment-matched GP to a finite set of noisy (independent) observations,

which can then be fitted with an RBF GP again, and so on until the end. The proposed method works very well for irregularly-sampled MNIST, and fairly well for the PhysioNet2012 data set for ICU mortality.

# Comments

The paper references related work, makes a meaningful contribution, and I think the empirical methodology is sound. Thus, I think it should be accepted. As negatives, there are a few small issues with the paper, and it is hard to follow.

## Central limit theorem

The CLT argument referenced in section 4.6, only works if all the elements of M are equal. (As a counterexample: suppose only one column of M is 1, and all the other ones are 0. Then, if the corresponding input process is non-Gaussian, all the outputs will be non-Gaussian).

In fact, the argument in the appendix forgoes coefficients for every $g_c$ altogether. It is correct, but it is not the same object in section 4.6.

I don't buy your CLT argument, but I think the rest of the paper is still great without it. Simply, you're doing a moment-matching approximation, like Expectation Propagation (Minka, 2001b).

I would consider dropping the CLT claims. Or, you could argue that in practice the elements of M are similar.

You could also argue that the elements of M are close to independent with mean zero. If M is also fairly large, you can argue that $f$ is close to a GP, following the theoretical arguments from the infinitely wide NN literature (Matthews et al., 2018; or generalizations thereof such as Yang, 2019).

Minka, T. P. (2001). Expectation propagation for approximate Bayesian inference. UAI (pp. 362-369)
Matthews, A G de G, et al. (2018). Gaussian process behaviour in wide deep networks. https://arxiv.org/abs/1804.11271
Yang, G., (2019). http://papers.neurips.cc/paper/9186-wide-feedforward-or-recurrent-neural-networks-of-any-architecture-are-gaussian-processes

## Some experiment and figure clarification

In section 4.7 you talk about a collection of points ${x_i}_{i=1}^N$ to which you project the activation. Are these points the same for every layer? Are they always the points at which the network is currently being evaluated (I understand these are points for one input point, i.e. sampling locations for an MNIST image)?

For figure 2, how many test images do you use? Is it one, with varying number of sampling locations per image?

For Figure 3, how do you change the test-time resolution of a normal CNN, one that has discretely specified pixels? Do you linearly interpolate their values? This should be explained.

## Other clarification requests and suggestions

I found the paper hard to follow, and some things are unclear. This is understandable, because it uses mathematical tools unusual for an ML paper, but you can still make it easier for the reader. The rest of my review is some requests for clarity and writing suggestions.

*Taylor expansion above eq. 3:* it is an expansion around $x$ of $g(x + a)$, no? I believe stating this simple fact after "Verify by Taylor expansion [around $x$ of $g(x + a)$] would make it much less confusing.

What is the "[a, b]" operator at the top of page 5?

*Eq 9*:  I think $\nabla\nabla^T {\boldsymbol \Phi}({\boldsymbol \mu}, {\boldsymbol \Sigma})$ is supposed to represent the Hessian of ${\boldsymbol \Phi}({\boldsymbol \mu}, {\boldsymbol \Sigma})$, but I'm not sure. It's not quite standard notation and should be mentioned in the paragraph above.

In section 4.7, when projecting the current activations to a set of points, you lose the covariance. It's good that you comment on this in Appendix H, and it would be better if you pointed the reader to it.

I think specifying that you use Adam in section 4.8 is helpful to the reader and doesn't take much space.

Appendix F: $3 \cdot 10^{-3}$, not $310^{-3}$.

Are you going to release the code?
review_title: Good, enjoyable paper; flimsy CLT; hard to follow

final_score: rating: 7: Good paper, accept
review_content: This paper is way out of my comfort zone so I can only provide very general high level comments.

First, the motivation for the paper seems very clear. For deep learning to work, we often have to discretize, and have to do so at lower resolution, which induces errors we typically ignore.

Second, the architecture is well described and illustrated. 

Some references (e.g. p1 Cockayne 2019) are in the wrong format. 

I didn't check the math in all details, but the three experiments with superpixel resolution, generalization to new resultions and and irregular time series appear convincing.
review_title: Probabilistic numerics inspired DNN using GPs

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This work presents an uncertainty aware continuous convolutional layers for learning from continuous signals like time series/images. This work is most useful in the setting of irregularly sampled data. Gaussian processes (GP) are used to represent the irregularly sampled input. The proposed continuous convolutional layers can be directly applied to input Gaussian Process in a closed form, which subsequently outputs another GP with transformed mean and variance. Finally, the continuous convolutional layers are parameterized in terms of the flow of a PDE. The use of GP for feature representation and the ability of continuous convolutional layers to take GP as input provides the model with the ability to propagate uncertainty.

Positives:
1. The problem addressed is very important in many domains. 
2. The proposed approach is novel for learning from irregularly sampled data.
3. Empirical results show that proposed model performs well.  

Concerns:
1. Given the time complexity of Gaussian processes, could the authors comment on the run-time of the proposed method as compared to standard CNN?
2. This work is missing comparisons with one of the recent work focusing on the same problem (Rubanova et al, 2019). It would be interesting to also compare the run-time of both these methods. 
3. In Section 2 para 3, the authors mention that Gaussian process adapter (Li and Marlin, 2016) operates on a deterministic signal
on a regular grid and cannot reason probabilistically about discretization errors. I don't think this is true. Infact, GP adapter is an uncertainty aware classification framework which has the ability to pass the uncertainty through to classification framework similar to the proposed approach. This makes these two approach more similar in what they are trying to achieve and I would expect the authors to compare the performance of these methods.
5. The authors repeatedly discuss that their approach does not require any discretization unlike some the previous approaches based on GP(mentioned in Section 2). But in Section 4.7, they mention that their approach requires to evaluate Gaussian Process at a set of points which is similar to what was done in Li and Marlin(2016). Could the authors clarify this?

Additional Comments:
1. Could the authors comment on the possibility of using the proposed approach for interpolation in irregularly sampled time series?

I would be happy to increase the score once my concerns are addressed.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: SUMMARY:

The paper considers the problem of using CNNs on an irregularly sampled grid. It proposes to incorporate the numerical uncertainty related to the disretisation of the domain using an approach inspired by probabilistic numerics. This involves defining continuous convolutional layers as affine transformations of the input GP and use rectified GPs to include the nonlinearity in the mapping. The resulting non-Gaussian distribution is then approximated with another GP in each layer. These are then approximated with a GP with an RBF kernel using Monte Carlo to produce observations with input dependent noise. The model is trained using MAP estimation.


##################################################################

REASON FOR SCORE:

I like the motivation for this work and the paper was generally a pleasure to read. Naturally, many technical challenges arise when trying to approach this problem from a probabilistic perspective; they are discussed in a fair bit of detail and generally seem reasonable. However, I still have a number of questions that are listed below that would help me better understand the main ideas and contributions.


##################################################################

PROS:
+ The motivation for the work seems sound.
+ The main technical contribution seems to be the definition of a continuous convolution on GPs with an RBF kernel. This is presented clearly and in a fair bit of detail.
+ The writing quality is very good. The appendix was very informative as well.


##################################################################

CONS:
- The paper makes quite a few approximations (which mostly make sense) but doesn't give a thorough discussion of the effect of these choices.
- The paper doesn't seem to make full use of the setup (see questions below).
- I found Fig.2, especially 1 and 3, hard to interpret and the figure caption wasn't very informative. It appears from the left figure that the mean is very smooth in all layers. Is that desirable? In the right figure, what is gt on the y-axis?


##################################################################

QUESTIONS and COMMENTS:
- My understanding is that PN approaches are typically used for tasks like, (a) to impose specific priors, (b) to improve the design of the problem or (c) to learn something about the implicit priors in established deterministic methods.
It would be interesting to think about these in the context of your model. For example, (a) is it reasonable to use a GP with a smooth kernel as an interpolant of the inputs? If the kernel choice was not limited by the analytic tractability, would you still choose an RBF kernel? For (b), could you use the model to decide which points in the input domain are the most informative for the end task? I suppose that you would need the full posterior distribution rather than just a point estimate, which would be very tricky given the multi-layer setup. Also, given that your approach seems to outperform existing (deterministic?) models or some tasks, how could you change the existing models to gain a similar advantage? 

- It would be interesting to know what is the effect of the various approximations, for example, the Gaussian approximation in 4.6 and the MC in 4.7, on the final task.

- The CLT argument in Appendix I is not very clear to me. The Lyapunov CLT holds in cases of RVs being independent, but not necessarily identically distributed. Is it correct that in your case the RVs are weakly-dependent and identically distributed? What is c in the inequality |c - c'| >> 1? More generally, is the argument in this section some well-known result? It shares some ideas with the argument of Neal [1996] of an infinite limit of NNs being equivalent to GPs but it would be good to either give a more thorough discussion of this argument or provide links to related results in the literature.

- It would be informative to discuss the computational and memory complexity of the proposed approach and other related questions, e.g. the number of samples in the MC steps.

- I'm a little surprised that removing the noise gives worse results in your MNIST experiment. I would guess that this may be more related to some optimisation phenomenon than to actual model. For example, if instead of using the data-dependent noise parameter as described in Sec. 4.7 you used a spherical Gaussian noise with fixed sigma for all inputs, would that give better results than discarding the noise entirely? And how would it compare with your data-dependent noise? Using spherical Gaussian noise relates with some of the practices in DGP models where the input to each layer is assumed to be a noisy version of the outputs of the previous layer. 

- Related literature: It might be interesting to compare to some DGP models that also use convolutions, e.g. Deep Gaussian Processes with Convolutional Kernels [2018] by Kumar et al and Deep convolutional Gaussian processes [2019] by Blomqvist et al. 

##################################################################

MINOR:

- p. 1 - Probalistic Numeric
- p. 6 - the number the number
- p. 15 - infinitesmal
- p. 18 - many many observations
- p. 19 - a variant a variant
review_title: Promising development in probabilistic convolutional models

Meta-review: This is a fairly technical paper bridging deep learning with uncertainty propagation in computations (i.e. probabilistic numerics). It is well structured, but it could benefit from further improvements in readability given that there are only very few researchers that are experts in all sub-domains associated with this work. Given the above, as well as low overall confidence by the reviewers, I attempted a more thorough reading of the paper (even if not an expert myself), and I was also happy to see that the discussion clarified important points. Overall, the idea is novel, convincing and seems well executed, with good results. The technical advancements needed to make the idea work are fairly complicated and are appreciated as contributions, because they are expected to be useful in other applications too (beyond irregular sampled data) where uncertainty propagation matters.
Decision: Accept (Poster)