Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper presents two models extended from a meta-presentation of neural networks (Karaletsos et al. 2018) in which neural network weights are constructed hierarchically from latent variables associated with each layer. The mappings from latent space to weight space are used Gaussian Processes instead of neural networks.

The proposed models include (1) MetaGP which directly replaces neural network assumption by Gaussian process prior and (2) MetaGP with contextual information which further takes into account input information via performing the multiplication between the kernel function over latent space and kernel function over input space.

Variational inference is followed by the pseudo-inducing point approach.

Experiments are conducted in both toy data sets and benchmark data sets, demonstrating several points i.e. uncertainty quantification, inductive bias. 

Pros:
The paper is clearly written.
Introducing inductive bias or functional regularization for neural network
Interesting capability of measuring the uncertainty of out-of-sample data. This can be one of the reasons that the method performed well in active learning.

Cons:
The approach is incremental or not-so-novel in terms of meta-representation for neural networks.

Comments and questions:
The prior distribution for latent variable $z$ is not specified. I assume the prior is independent Gaussian.
$z$ is unknown beforehand. How are inducing locations for latent variable $z$ initialized?
 Do you think that there is a connection between contextual metaGP and residual nets? Can skip connections from the input layer to certain layers be considered to be similar to the idea of incorporate input kernel in the paper?
Can you comment on the convergence of the estimation of the last term in the variational bound? The MCMC takes two stages of stochasticity: (1) sample $z_k$ and $V_k$ and (2) then estimate $F_k$ using reparameterization. This can make the convergence slow (https://arxiv.org/abs/1709.06181). 
Minor: a missing period in Sec. 6.3 “quadratic kernel In this example”
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: **Summary**: This paper proposes a hierarchical Bayesian approach to hyper-networks by placing a Gaussian process prior over the latent representation for each weight. A stochastic variational inference scheme is then proposed to infer the posterior over both the Gaussian process and the weights themselves. Experiments are performed on toy regression, classification, (edit: post rebuttal) and transfer learning tasks, as well as an uncertainty quantification experiment on MNIST.

post rebuttal (noticed recently): Many apologies for updating the review the day before the deadline; however, I recently remembered that Kronecker inference is often used in variational methods - particularly within the vein of literature of deep kernel learning. Indeed, structure exploiting SVI was proposed in Stochastic Variational Deep Kernel Learning, https://arxiv.org/pdf/1611.00336.pdf, and this method is currently the default in Gpytorch: https://github.com/cornellius-gp/gpytorch/tree/master/examples/08_Deep_Kernel_Learning .
Furthermore, Kronecker inference for non-Gaussian likelihoods for Laplace approximations was proposed back in 2015: http://proceedings.mlr.press/v37/flaxman15.pdf. 
I am not updating my score because it would be unfair; however, the record should be set somewhat straight here.

post rebuttal: Thank you for the many clarifications and detailed responses. I'm now satisfied with their many changes and and tend to accept this paper despite the experimental results being somewhat limited. I would really encourage the authors to fix the color schemes (please less black and more brighter colors) on their decision boundaries plots however. 

**tldr**: While I appreciate the concept of this paper, I tend to reject this paper because I find the experimental results to be on too small scale of datasets. Specifically, I would like to see either a larger scale problem being solved with this kind of approach or a tough to model applied problem that is solved with this approach.

**Originality**: As far as I can tell, this seems to be a novel approach to hyper-networks. Neural processes (Garnelo et al; 2018) propose a somewhat similar approach to training – with a latent process over some stored weight space. However, even that is quite distinct from the method proposed in this paper, and I tend to prefer this approach. 

**Quality**: I really appreciate the merging of neural network and Gaussian process methods; however, tragically, I do wonder if the proposed approach combines the worst of both worlds – the necessity of architecture search for neural networks with the choice of kernel function (as illustrated in Figure 5). 
If the method is truly kernel dependent, is it also architecture dependent? That is, is it robust to different settings of nonlinearities and depths?

Active learning experiment: While I appreciate the comparison here, it seems like here standard HMC should be trainable over well-designed priors on these architectures. So why not include a comparison instead of just MFVI?

**Significance**: Unfortunately, I think that the experiments section is just a bit too limited to warrant acceptance right now. This is despite the fact that I really do appreciate the thoroughness and thoughtfulness of the experiments as they are. 

Specifically, in Section 6.2 why is the metaGP prior only applied to the last layer of the network? If as I suspect, it is due to the complexity and difficulty of inference, that makes the method doubly tough to use in practice. With that being said, to only have experiments on the last layer implies that one should compare to Bayesian logistic regression and linear regression on the last layer of neural networks (e.g Perrone et al, 2018 and Riquelme et al, 2018). Experiments with other methods that combine Gaussian processes with representations on the final layer (e.g. Wilson et al, 2015) are also probably worth running. 

Figure 4 is a very well-done experiment, if a bit tough to read. I’d suggest that the out of distribution examples get their own figure, with the in distribution examples going into the appendix. I’d also suggest computing the expected calibration error (Naeini et al, 2015) for in and out of distribution examples on the test sets for both MNIST and K-MNIST in order to have quantitative results on the entire test set. 

To recommend acceptance, I’d really have to see experiments on either a CIFAR sized dataset for classification or a larger scale regression experiment. A larger dataset on either transfer learning (after all you do have a meta-representation over functions that the NN can learn), a larger active learning experiment, or semi-supervised learning. 

**Clarity**: Overall, the paper is well-written and mostly easy to follow. The meat of the paper is found in Section 4, which I found a bit difficult to follow. 

(edit: post rebuttal. This concern is somewhat resolved due to the field not being well developed in this area, although it is a useful place to possibly extend the method in the future.) My primary concern here is that the prior ends up becoming Kronecker structured (after Eq. 7), so it isn’t clear to my why dense matrices and dense variational bounds have to be derived in this setting. Can one not follow the lead of the Gaussian process literature (e.g. Saatci 2012, Wilson & Nickisch, 2015) to exploit the Kronecker structure here to make computation of the log likelihoods fast?
(edit: post rebuttal. This concern is somewhat resolved.) As a result, it’s not immediately clear to me why a diagonal approximation (Eq. 10) is even necessary? 
Furthermore, this may be a setting where iterative methods (e.g conjugate gradients and Lanczos decompositions as in Pleiss et al, 2018) for the predictive means and variances may shine and be fast.
I do agree that the approximation in Figure 2 does seem to be relatively accurate, although I would ask the authors to compute a relative error for that plot if possible. Additionally, what is the strange high off diagonal correlations in the marginal covariances?

(edit: post rebuttal. Thank you for the clarifications here.) Finally, I was a bit confused by the effect of adding the input dependent kernel in Section 3; this seems to make the weights much more complicated to model – now each data point has its own set of weights and therefore, we might have to store considerably more weight matrices over time. Could the authors perform a set of experiments showing the necessity of this kernel matrix in the rebuttal?

**Minor Comments**: 
-	Above Eq. 9, “splitted” should be split.
-	Figure 3: could the data points be plotted in a brighter fashion? On a dark background, they are quick tough to see. Additionally, what is the difference between the two levels of classification plots?


References:

Naeini, et al. Obtaining Well Calibrated Probabilities by Bayesian Binning, AAAI, 2015. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4410090/

Perrone, V, et al. Scalable Hyperparameter Transfer Learning, NeurIPS, 2018. http://papers.nips.cc/paper/7917-scalable-hyperparameter-transfer-learning

Pleiss, G, et al. Constant Time Predictive Distributions for Gaussian Processes, ICML, 2018. https://arxiv.org/abs/1803.06058

Riquelme, C, Tucker, G, Snoek, J. Deep Bayesian Bandits Showdown, ICLR, 2018. https://arxiv.org/abs/1802.09127

Saatci, Y. Scalable Inference for Structured Gaussian process models, PhD Thesis, U. of Cambridge, 2011. http://mlg.eng.cam.ac.uk/pub/pdf/Saa11.pdf

Wilson, AG and Nickisch, H. Kernel Interpolation for Scalable Structured Gaussian Processes, ICML, 2015. http://proceedings.mlr.press/v37/wilson15.pdf

Wilson, AG, et al. Deep Kernel Learning, AISTATS, 2015. https://arxiv.org/abs/1511.02222

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes an improvement over Probabilistic meta-representations of neural networks by replacing the NN parameterization of the network weights given latent variables by a probabilistic distribution whose mean is distributed by GP. Inference of the induced hierarchical model is achieved by variational inference and various
approximations when needed.

Comments:

1. The authors claim that the proposed method aims to increase the robustness in the small data settings
and improve its out-of-sample uncertainty estimates. The second part is well justified. 
Could the author elaborate further on the high level intuition behind the first part?

2. What exactly is the gain obtained by replacing a NN parameterization with a GP parameterization?
It seems like the proposed method gains the ability to model uncertainty, but potentially incurs performance trade-off 
from a series of approximation. 

3. Following from comment #2, I am a bit surprised by the lack of comparison against the work of 
Karaletsos although this work was built on top of it. I am interested to understand the mentioned
trade-off in practice.

4. Regarding the practical consideration in Eq.(11), I think it kind of defeats the purpose of setting up
the latent variables so that "weights in a layer and across layers are explicitly correlated in the 
modelling stage" (Section 2.2). Do all experiments presented in the paper employ this practical 
consideration?

5. For the text before Eq. (8) should the inducing inputs be xu instead of zu? It seems like a systematic
typo here because in the formulation for the lower bound it becomes p(u|xu) again instead of p(u|zu)

6. Is Kuu in Eq.(9) computed by taking Kronecker product of K_in, K_out and K(xu,xu) or just K(xu,xu).
I am under the impression that it is the former, in which case taking the inversion is costly (cubic in the number
of latent variables). This would not permit large NN anyway, which kind of defeats the purpose. It explains the choice 
of small architecture in your experiments as well. 

Overall comment:

I think the paper presents an interesting idea but I have questions regarding its practical significance as highlighted in my specific comments above. I hope the authors would clarify these so I can converge on a final rating.

review_title: Official Blind Review #1

Meta-review: The authors propose an approach to Bayesian deep learning, by representing neural network weights as latent variables mapped through a Kronecker factored Gaussian process. The ideas have merit and are well-motivated. Reviewers were primarily concerned by the experimental validation, and lack of discussion and comparisons with related work. After the rebuttal, reviewers still expressed concern regarding both points, with no reviewer championing the work.

One reviewer writes: "I have read the authors' rebuttal. I still have reservation regarding the gain of a GP over an NN in my original review and I do not think the authors have addressed this very convincingly -- while I agree that in general, sparse GP can match the performance of GP with a sufficiently large number of inducing inputs, the proposed method also incurs extra approximations so arguing for the advantage of the proposed method in term of the accurate approximate inference of sparse GP seems problematic."

Another reviewer points out that the comment in the author rebuttal about Kronecker factored methods (Saatci, 2011) for non-Gaussian likelihoods and with variational inference being an open question is not accurate: SV-DKL (https://arxiv.org/abs/1611.00336) and other approaches (http://proceedings.mlr.press/v37/flaxman15.pdf) were specifically designed to address this question, and are implemented in popular packages. Moreover, there is highly relevant additional work on latent variable representations for neural network weights, inducing priors on p(w) through p(z), which is not discussed or compared against (https://arxiv.org/abs/1811.07006, https://arxiv.org/abs/1907.07504). The revision only includes a minor consideration of DKL in the appendix. 

While the ideas in the paper are promising, and the generally thoughtful exchanges were appreciated, there is clearly related work that should be discussed in the main text, with appropriate comparisons. With reviewers expressing additional reservations after rebuttal, and the lack of a clear champion, the paper would benefit from significant revisions in these directions. 

Note: In the text, it says:
"However, obtaining p(w|D) and p(D) exactly is intractable when N is large or when the network is large and as such, approximation methods are often required."
One cannot exactly obtain p(D), or the predictive distribution, regardless of N or the size of the network; exact inference is intractable because the relevant integrals cannot be expressed in closed form, since the parameters are mapped through non-linearities, in addition to typically non-Gaussian likelihoods.
Decision: Reject