Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes to blend the strengths of deep learning and the clear modeling capabilities of GPs by using a composite kernel that combines a kernel implicitly defined by a neural network with a second kernel function chosen to model known properties. Technically, the authors approximate the resultant GP by combining a deep network and an efficient mapping based on the Nyström approximation. Results on both synthetic and real-world data sets show the superior performance and flexibility of the proposal.
strengths_and_weaknesses: Strengths
- The paper proposes a practical method to incorporate prior knowledge into NNs. It offers an effective way for processing data with multiple sources of information where some explicit priors can be introduced in the form of GPs.
- The paper establishes a way to integrate NNs and GPs efficiently and train them jointly.
- The empirical results are interesting and insightful.


Weaknesses
- Assumption 1 seems to apply to NN-GP kernels but does not apply to the approximate NN-GP kernels corresponding to finite-width NNs.
It does not apply to typical kernels like RBF/polynomial ones as well. Can you clarify its validity theoretically/empirically?

- In the case of an NN (whose parameters are $\theta^{(1)}$) plus a classic kernel (whose parameters are $\theta^{(2)}$), the learning is to find a good distribution $p(\theta^{(1)})$ and a point-estimate $\theta^{(2)}$? If so, can you clarify how you parameterize $p(\theta^{(1)})$? (E.g., a variational BNN with Gaussian variational whose mean is fixed as 0?) If not, and if you just learning a point-estimate $\theta^{(1)}$, Lemma 1 would not apply and Theorem 2 does not hold.

- The loss function for training is just the MSE error? Can the framework be extended to handle classification problems? It seems that it is non-trivial as the output should only be 1-D. 

- How to pre-define the integrating points? Can they be updated by the training signals?

- How does the proposal compare to the recent work NeuralEF?

NeuralEF: Deconstructing Kernels by Deep Neural Networks. ICML 2022.
questions: See Weaknesses
limitations: the authors have adequately addressed the limitations
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors introduce a neural network construction that forms predictions by computing inner products between a hidden representation generated by a neural network and a kernel-derived basis of functions.  This construction is motivated by a desire to combine the flexibility of neural networks with the domain-specific knowledge that can be encoded in kernels (e.g. seasonality). The properties of the resulting method are explored on three data sets.
strengths_and_weaknesses: The main strength of this submission, as I see it, is that one of the empirical comparisons includes a remote sensing data set, which would seem to be a rich test bed for these kinds of methods---though it should be noted that accessing this data set requires a PlanetScope license.

Unfortunately, the weaknesses of this submission outweigh the strengths by a good margin. The novelty and empirical validation presented would seem to be more in line with a workshop submission than a full paper.
1. The suggested method is more trivial than not and I am somewhat skeptical that very similar and/or identical methods do not already exist in the literature. However, even if we take it for granted that nothing like this has been done before, the empirical comparisons are too shallow, see below.
2. The method isn't specified in sufficient detail. We are told (line 143ff) that "the parameters of both the NN and the kernel function are learned via gradient-based optimization methods." OK but with what objective function? Mean squared error for regression tasks? Something else? Does the objective function matter? 
3. The number/variety of benchmark algorithms used in experiments is inadequate. What about a MLP that is given access to cyclical/fourier features in the time domain in the experiments in Sec. 5.3? What about various neural processes, e.g. reference [1] below? What about attention-based mechanisms? The omission of the latter is particularly troubling given the empirical success of attention-based methods and the similarity of attention-based mechanisms with your inner product construction. Given the simplicity of the proposed construction, a thorough empirical assessment (using a variety of datasets and benchmark methods) is absolutely crucial to convince the reader that the proposed method is useful in practice. In general the appendix reveals very little information about the MLPs used in the experiments. How many hidden units? How were these trained? etc. The proposed method may be interesting, but these toy experiments do not convince. 

References
1. "Convolutional Conditional Neural Processes,"
Jonathan Gordon, Wessel P. Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, Richard E. Turner
2. "Attention Is All You Need,"
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
questions: - For M>2 the authors suggest using a "chained inner product" to form predictions (Appendix A). The authors also state (line 295ff) that "A major limitation of ICK lies in our method of combining latent representations as the nature of inner product (i.e. the effect of multiplying small numbers) may cause vanishing gradient problems when we have a large number of sources of information (i.e. M is large)." Have you tried M>3? When do you expect ICK to break down?
- What happens when Assumption 1 (line 158ff) does not hold? Can you test this experimentally with an artificial dataset that breaks this assumption?
limitations: The authors do not discuss any potential negative societal impacts of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a way to combine prior knowledge inside deep networks. It assumes data comes from different sources (at least one low-dimensional source and one high-dimensional source), and prior knowledge is known about the low-dimensional one (e.g., seasonality). They build a composite kernel where the low-dimensional data is processed with a user-defined kernel, while the high-dimensional one with a neural network. The former is then processed though kernel approximation methods to obtain a latent representation which is multiplied by the embedding of the neural network to get the final prediction. They perform several experiments ranging from a synthetic dataset to a realistic use case in remote sensing.
strengths_and_weaknesses: The idea is simple but (as far as I know, as I am not an expert in neural Gaussian Processes) novel. The paper is easy to read, the technique sound, and the experiments are interesting and relatively clear in the benefit of the method.
questions: - I think computational time should be discussed more. The Cholesky decomposition is cubic in the number of pseudo-inputs, and the overhead required by differentiating through it is unclear to me.
- "Incorporating Prior Knowledge" is a pretty strong title and motivation for the paper, since the technique is feasible only in the very specific setting described in the summary. It is impossible (as far as I understand) to include generic prior information if it is relative to the high-dimensional data (e.g., an image). I think the claims in the paper should be downsized accordingly.
- Since the kernel is included in the autodiff process, I wonder whether it is also possible to optimize some of its parameters (e.g., the seasonality) automatically. Have the authors considered this point?
limitations: NA
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose ICK, a regressor that can naturally incorporate priors over low-dimensional subsets of the input. A high-dimensional input is processed with a neural network to produce a latent vector, and the low-dimensional input produces a latent vector of the same size via a Gaussian process (GP), where the kernel function incorporates the known priors. The GP is efficiently integrated using the Nystrom method and Cholesky decomposition.
strengths_and_weaknesses: Strengths
The approach is an elegant combination of GPs and neural networks for incorporating useful priors, supported by a number of experiments.
The paper is well-written and the figures are very clear.
Re: vanishing gradients as a limitation. I actually think your approach improves on the naive approach wrt vanishing gradients! Normally a network may ignore low-dimensional inputs even if they are informative, but your projection of both high-dimensional and low-dimensional data sources into a common latent space encourages them to be treated more equally.

Weaknesses
One of the nice properties of GPs is that they can capture uncertainty in the prediction. However, the toy experiment suggests that this uncertainty is perhaps lost or not well calibrated. For example, I'm not sure that reverting to 0 (Fig. 3 bottom) counts as "failing gracefully" when the standard deviation is so small. Why is it confident in its bad prediction, and is there a way to mitigate this? It would be nice to see an experiment/metric that also demonstrates that your method matches distributions well.
It may be helpful to discuss what kinds of priors are amenable to being represented as the kernel function of a GP. It seems that this work focuses on continuous scalar variables where the prior represents periodicity, and the experiments are a little limited in this respect. Discuss some other types of priors and their corresponding kernels, and if possible add an experiment along this line.

questions: As mentioned in related work, there are at least 1-2 other methods that equip NNs with priors using GPs. Can you compare with them or at least explain when/why they would be less effective than your approach?
It's not entirely clear to me what integral is being approximated with Nystrom quadrature. Is it the expectation in (10)? Might be helpful to briefly mention somewhere in 4.2.1 when you talk about integrating points.
Consider rescaling the numbers in Table 1 so there aren't so many decimal places.
Perhaps the conjugate gradient method is a feasible alternative to Cholesky decomposition if p needs to be larger? For future consideration.
Line 90 chose -> choose
limitations: Yes. Assumption 1 seems reasonable to me, but the authors should try to characterize the cases where it doesn't hold. Perhaps a rigorous condition is hard to derive, but is there any intuitive argument that the cases where it doesn't hold are ill-behaved?
review_title: null

Meta-review: metareview: The submission considers fusing prior knowledge into neural networks by *modulating* the learnt features. The modulation is either additive or multiplicative using another set of features outputted by a kernel-based mapping with linear/periodic kernels. The method is akin to using composite kernels (or combining kernels) in the GP literature and is tested on several regression benchmarks.

The reviewers acknowledged the method is simple and seems to give a desirable performance when the prior knowledge is known. I (AC) read the paper and have several concerns (some of which have already been raised by one or more reviewers):
- the clarity around the connection to composite GP kernels could be improved (there was also a question about the goal of this connection and the authors stated that it is to motivate the proposed approach),
- many desirable properties of GPs (hyperparameter learning using the marginal likelihood, predictive uncertainty) seem to be lost since the objective function is only MSE/MAE and the parameters are not treated probabilistically [the authors promised to consider this in the next iteration].
- limited experiments to show the capability of the ICK framework beyond two data sources, and for various forms of prior knowledge beyond seasonality and the PM 2.5 forecasting task. Some further analyses of the forecasting results + errorbars would be appreciated.

Despite the simplicity of the approach, for the above reasons, I feel that the submission is not ready for publication. I hope, however, to see an updated version published at a conference soon. The reviews are already fairly positive and the discussions here could be useful for polishing up the experiments and writing.
 
confidence: Certain
Decision: Reject