Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Neal (1994) showed that a one hidden layer Bayesian neural network, under certain conditions, converges to a Gaussian process as the number of hidden units approaches infinity. Neal (1994) and Williams (1997) derive the resulting kernel functions for such Gaussian processes when the neural networks have certain transfer functions.

Similarly, the authors show an analogous result for deep neural networks with multiple hidden layers and an infinite number of hidden units per layer, and show the form of the resulting kernel functions. For certain transfer functions, the authors perform a numerical integration to compute the resulting kernels. They perform experiments on MNIST and CIFAR-10, doing classification by scaled regression. 

Overall, the work is an interesting read, and a nice follow-up to Neal’s earlier observations about 1 hidden layer neural networks. It combines several insights into a nice narrative about infinite Bayesian deep networks. However, the practical utility, significance, and novelty of this work -- in its current form -- are questionable, and the related work sections, analysis, and experiments should be significantly extended. 


In detail:

(1) This paper misses some obvious connections and references, such as 
* Krauth et. al (2017): “Exploring the capabilities and limitations of Gaussian process models” for recursive kernels with GPs.
* Hazzan & Jakkola (2015): “Steps Toward Deep Kernel Methods from Infinite Neural Networks” for GPs corresponding to NNs with more than one hidden layer.
* The growing body of work on deep kernel learning, which “combines the inductive biases and representation learning abilities of deep neural networks with the non-parametric flexibility of Gaussian processes”. E.g.: (i) “Deep Kernel Learning” (AISTATS 2016); (ii) “Stochastic Variational Deep Kernel Learning” (NIPS 2016); (iii) “Learning Scalable Deep Kernels with Recurrent Structure” (JMLR 2017). 

These works should be discussed in the text.

(2) Moreover, as the authors rightly point out, covariance functions of the form used in (4) have already been proposed. It seems the novelty here is mainly the empirical exploration (will return to this later), and numerical integration for various activation functions. That is perfectly fine -- and this work is still valuable. However, the statement “recently, kernel functions for multi-layer random neural networks have been developed, but only outside of a Bayesian framework” is incorrect. For example, Hazzan & Jakkola (2015) in “Steps Toward Deep Kernel Methods from Infinite Neural Networks” consider GP constructions with more than one hidden layer. Thus the novelty of this aspect of the paper is overstated. 

See also comment [*] later on the presentation. In any case, the derivation for computing the covariance function (4) of a multi-layer network is a very simple reapplication of the procedure in Neal (1994). What is less trivial is estimating (4) for various activations, and that seems to the major methodological contribution. 

Also note that multidimensional CLT here is glossed over. It’s actually really unclear whether the final limit will converge to a multidimensional Gaussian with that kernel without stronger conditions.  This derivation should be treated more thoroughly and carefully.

(3) Most importantly, in this derivation, we see that the kernels lose the interesting representations that come from depth in deep neural networks. Indeed, Neal himself says that in the multi-output settings, all the outputs become uncorrelated. Multi-layer representations are mostly interesting because each layer shares hidden basis functions. Here, the sharing is essentially meaningless, because the variance of the weights in this derivation shrinks to zero. 
In Neal’s case, the method was explored for single output regression, where the fact that we lose this sharing of basis functions may not be so restrictive. However, these assumptions are very constraining for multi-output classification and also interesting multi-output regressions.

[*]: Generally, in reading the abstract and introduction, we get the impression that this work somehow allows us to use really deep and infinitely wide neural networks as Gaussian processes, and even without the pain of training these networks. “Deep neural networks without training deep networks”. This is not an accurate portrayal. The very title “Deep neural networks as Gaussian processes” is misleading, since it’s not really the deep neural networks that we know and love. In fact, you lose valuable structure when you take these limits, and what you get is very different than a standard deep neural network. In this sense, the presentation should be re-worked.

(4) Moreover, neural networks are mostly interesting because they learn the representation. To do something similar with GPs, we would need to learn the kernel. But here, essentially no kernel learning is happening. The kernel is fixed. 

(5) Given the above considerations, there is great importance in understanding the practical utility of the proposed approach through a detailed empirical evaluation. In other words, how structured is this prior and does it really give us some of the interesting properties of deep neural networks, or is it mostly a cute mathematical trick?  

Unfortunately, the empirical evaluation is very preliminary, and provides no reassurance that this approach will have any practical relevance:
(i) Directly performing regression on classification problems is very heuristic and unnecessary.
(ii) Given the loss of dependence between neurons in this approach, it makes sense to first explore this method on single output regression, where we will likely get the best idea of its useful properties and advantages. 
(iii) The results on CIFAR10 are very poor. We don’t need to see SOTA performance to get some useful insights in comparing for example parametric vs non-parametric, but 40% more error than SOTA makes it very hard to say whether any of the observed patterns hold weight for more competitive architectural choices. 

A few more minor comments:
(i) How are you training a GP exactly on 50k training points? Even storing a 50k x 50k matrix requires about 20GB of RAM. Even with the best hardware, computing the marginal likelihood dozens of times to learn hyperparameters would be near impossible. What are the runtimes?
(ii) "One benefit in using the GP is due to its Bayesian nature, so that predictions have uncertainty estimates (Equation (9)).”  The main benefit of the GP is not the uncertainty in the predictions, but the marginal likelihood which is useful for kernel learning.
review_title: Interesting paper but preliminary

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper leverages how deep Bayesian NNs, in the limit of infinite width, are Gaussian processes (GPs). After characterizing the kernel function, this allows us to use the GP framework for prediction, model selection, uncertainty estimation, etc.


- Pros of this work

The paper provides a specific method to efficiently compute the covariance matrix of the equivalent GP and shows experimentally on CIFAR and MNIST the benefits of using the this GP as opposed to a finite-width non-Bayesian NN.

The provided phase analysis and its relation to the depth of the network is also very interesting.

Both are useful contributions as long as deep wide Bayesian NNs are concerned. A different question is whether that regime is actually useful.


- Cons of this work

Although this work introduces a new GP covariance function inspired by deep wide NNs, I am unconvinced of the usefulness of this regime for the cases in which deep learning is useful. 

For instance, looking at the experiments, we can see that on MNIST-50k (the one with most data, and therefore, the one that best informs about the "true" underlying NN structure) the inferred depth is 1 for the GP and 2 for the NN, i.e., not deep. Similarly for CIFAR, where only up to depth 3 is used. None of these results beat state-of-the-art deep NNs.

Also, the results about the phase structure show how increased depth makes the parameter regime in which these networks work more and more constrained. 

In [1], it is argued that kernel machines with fixed kernels do not learn a hierarchical representation. And such representation is generally regarded as essential for the success of deep learning. 

My impression is that the present line of work will not be relevant for deep learning and will not beat state-of-the-art results because of the lack of a structured prior. In that sense, to me this work is more of a negative result informing that to be successful, deep Bayesian NNs should not be wide and should have more structure to avoid reaching the GP regime.


- Other comments:

In Fig. 5, use a consistent naming for the axes (bias and variances).

In Fig. 1, I didn't find the meaning of the acronym NN with no specified width.

Does the unit norm normalization used to construct the covariance disallow ARD input selection?


[1] Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. The Curse of Dimensionality for Local Kernel Machines. 2005.
review_title: Interesting characterization of deep Bayesian NNs, but not the most interesting regime for deep learning.

final_score: rating: 7: Good paper, accept
review_content: This paper presents a new covariance function for Gaussian processes (GPs) that is equivalent to a Bayesian deep neural network with a Gaussian prior on the weights and an infinite width. As a result, exact Bayesian inference with a deep neural network can be solved with the standard GP machinery.


Pros:

The result highlights an interesting relationship between deep nets and Gaussian processes. (Although I am unsure about how much of the kernel design had already appeared outside of the GP literature.)

The paper is clear and very well written.

The analysis of the phases in the hyperparameter space is interesting and insightful. On the other hand, one of the great assets of GPs is the powerful way to tune their hyperparameters via maximisation of the marginal likelihood but the authors have left this for future work!


Cons:

Although the computational complexity of computing the covariance matrix is given, no actual computational times are reported in the article.

I suggest using the same axis limits for all subplots in Figure 3.
review_title: Well written and insightful

Meta-review: This paper presents several theoretical results linking deep, wide neural networks to GPs.  It even includes illuminating experiments.

Many of the results were already developed in earlier works. However, many at ICLR may be unaware of these links, and we hope this paper will contribute to the discussion.

Decision: Accept (Poster)