Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: ### Summary
The authors propose a rule for neural network (NN) initialization, which takes into account input data. 

They suppose that weights and biases of a NN are randomly drawn resp. from $\mathcal{N}(0, \sigma_w^2 / N)$ and $\mathcal{N}(0, \sigma_b^2)$, where $N$ is the number of inputs. Then, they are able to compute *explicitly* the covariance matrix of the corresponding Gaussian Process. Since an explicit result is needed, they concentrate on one-layer NNs.

They use their explicit formula for the covariance matrix to compute the likelihood of the data, given $\sigma_w^2$ and $\sigma_b^2$. Therefore, they are able to select the best pair $(\sigma_w^2, \sigma_b^2)$ according to data, i.e., maximizing the likelihood.

### Clarity
I did not understand the experimental setup presented in Section 2.5. For instance, the train/test location are supposed to lie in the interval $[0, 1]$, but the test location points lie in $[0, 2]$ in the graphs.
Besides, all the computation of the covariance should be put in appendix.

### Significance
Since the paper relies on an explicit computation in one-layer NNs, the presented method has a very low significance. At least, the authors should propose an application in deeper NNs, even by making strong approximations. As such, no clue about any generalization is provided.

Edit:
### Rebuttal
I did read the authors' rebuttal, and the main issue, i.e. the significance, has not been addressed. I cannot take into account the new experiments, since they are not in the paper. Anyway, an experiment with a 2-layer network would be a significant modification of the present paper, which would be not acceptable during the rebuttal phase.
review_title: Finding initialization rules through the study of Gaussian Processes: one-layer case

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### UPDATE:

Per my response in the thread, I appreciate the authors' replies and updates, but I am keeping my score because
- Even with the new 2-layer results, I find this experimental setting still too limited;
- Even within the conducted experimental setting, the benefit of likelihood-guided initialization over He init is not robust (notably on small dataset sizes, which is contrary to the expectation that a good prior will be more beneficial in such cases).

### SUMMARY: 
The paper proposes to use the marginal likelihood of the Gaussian process (GP) corresponding to the infinite width Bayesian neural network (NN) to select hyper-parameters of the finite width, SGD-trained NN (instead of evaluating performance on the validation set). 

Experimentally, weight and bias variance in 1-hidden layer ReLU network are considered on MNIST, and on multiple training set sizes GP-likelihood-selected configuration slightly outperforms the default He-initialization, and slightly under-performs the best possible configurations. 

The paper also provides an alternative derivation of the ReLU kernel.


### REVIEW SUMMARY:
While I appreciate and support the question (**Q: Can we use NN-GP likelihood to select hyper-parameters of a finite SGD-trained NN?**) the paper raises, I believe the paper does not answer it (due to very limited experimental evidence), and raising the question itself is not sufficient for publication.


### PROS:
1. I believe the proposed idea is promising. 
2. Answering the question **Q** is certainly of interest to the research community. Unlike training and evaluating on a validation set, marginal likelihood of the NN-GP can be simply evaluated in closed form on a small subset of the training set, potentially making it a cheap proxy for validation performance.
2. The paper is clearly written and easy to follow.


### CONS:
1. My key concern: unfortunately experimental validation of the hypothesis is not adequate. After reading the paper I remain in the dark regarding whether I should use NN-GP likelihood for hyper-parameter search or not. Precisely:

	1. The idea is only evaluated on a single hidden layer, fully connected network, on MNIST (Table 1). 
    	1. MNIST is a very simple dataset, and a fully connected network is not well-suited for image classification. 
		2. The differences between test performance of best and worst configurations are within 1%, and differences between He-Init and NN-GP-guided configs are within 0.25%. 
		3. Notably, the likelihood-guided config performs worse than He-Init on the smallest training set size (10000). This is counter to the hypothesis that NN-GP likelihood evaluates the suitability of the NN prior, since we expect the quality of the prior to become more and more important as the training set becomes smaller. 
		4. Finally, the likelihood-guided config of (3.6, 0) is at the corner of the hyper-parameter grid, which makes it unclear what would be observed if the grid of weight variances was extended further. 

    Due to all of the above, I find this experiment unconvincing. The hypothesis should be evaluated on a hyper-parameter grid that spans a large range of model performances (vs fractions of a percent), one where likelihood is not maximized at an edge of the grid, as well as considering other datasets (e.g. CIFAR-10), other hyper-parameters (e.g. depth), and more appropriate architectures (e.g. CNNs for image classification tasks). Note that there are a many other nuances that would need to be addressed to properly answer the the question **Q** (e.g. comparing computational expense of NN-GP-based search to those of grid/random/Bayesian search, including tuning the additional meta-hyper-parameters (NN-GP training set size, diagonal regularization), discussion of the additional technical complexity and feasibility of evaluating the NN-GP on practical architectures, how wide the finite network has to be in practice etc). It would be OK to not touch on these in the same paper, and only provide a convincing answer to a more specific question like **"Q0: How well does NN-GP likelihood correlate with SGD-NN generalization?”**. Unfortunately, the paper doesn’t answer this either.
        
	2. The idea itself is fairly straightforward, not particularly motivated theoretically (e.g. is there a bound on the difference in performance between NN-GP and SGD-trained NNs, perhaps in some toy cases?), and arguably not novel, as it is raised in one of the referenced papers ([Deep Neural Networks as Gaussian Processes, Lee et al 2018](https://arxiv.org/abs/1711.00165), page 6, second to last paragraph before section 3). This is to highlight that I see most of the potential value of this paper in answering the question **Q/Q0** (vs raising it), hence putting so much importance on my prior point. This would not have been an issue for me if the question was answered.

### QUESTIONS:
1. Could you please provide data (if it is present) for Table 1 for training sizes of 1000, 3000, 5000? These appear in Figure 3, but not Table 1.
2. To tackle **Q0**, I encourage the authors to plot NN-GP likelihood vs test performance for all configurations that were considered (in addition to Table 1). To be clear, I’m afraid this would still not be sufficient to make a convincing case due to the limitations of the considered grid search discussed above.
3. I am not certain/convinced with the point Figure 3 is making: is it that training curves of He-init and NN-GP-guided become more similar with increasing training set size? If so, I find the effect to be too small to be convincing on Figure 3 (perhaps zooming in  / measuring numerically could help), but I am also not sure why this would be an interesting/important metric to look at (vs test accuracy, as measured in Table 1).

review_title: Raises an interesting question. Unfortunately, doesn't answer it.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

The paper considers the initialisation of fully connected neural networks with ReLUs.
By using the GP-one hidden layer neural network equivalence result, the paper considers using the equivalent GP and its marginal likelihood to decide the initial sampling parameters for the weights and biases. The paper first re-derives the covariance function of the GP corresponding to a single-hidden-layer ReLU network. To motivate the use of the marginal likelihood, the paper provides a simulation showing optimising the marginal likelihood on the data drawn from a GP arrives at hyperparameters that closely match the hypers used to generate the data. An experiment with neural network regression on the MNIST dataset shows the initialisation based on the GP marginal likelihood is promising.

Assessment:

Strengths:

1. The submission tackles an important problem in neural networks -- initialisation as despite a suite of techniques proposed for this task, any new technique that can consistently improve over the old ones in practice or provide more theoretical insights will be beneficial to the field. 

2. The topic of connecting (deep) neural networks and GPs and how to exploit this to improve neural network training is important. This submission shows this connection is useful for initialisation, as the marginal likelihood for GP regression can be computed in closed-form and for small dataset, optimising this objective is not costly.

Weaknesses:

3. novelty: The elements of the proposed approach are not novel. The covariance function for ReLU networks is not new and I don’t see the new insights from the new derivation compared to the work of Lee et al (for example: does this make things simpler when moving to deeper networks or a different activation function). The simulation study is useful, but not surprising since given enough data points the maximisers of the marginal likelihood should be the hyperparameters used to generate the data. 

4. practical significance: The experiment on the MNIST dataset is promising, however, it fell short of bringing the strengths (1 and 2) home. For example, it is not clear to me why a practitioner should choose the proposed approach which requires training (potentially expensive if the subset is large) over simpler approaches like He-init --- figure 3 shows no improvement over He-init and the accuracy results in table 1 doesn’t really show a clear difference between methods at the end. I’m not entirely sure I got the reasoning of the experimental set-up, e.g. why the training sets are subsets of the full MNIST, and what is the difference in terms of results between different training regimes? The task considered here is MNIST regression with a network with only one hidden layer. It is therefore very hard to justify the ‘near-optimal’ performance claimed early in the paper. It’d be good to show that the same approach (with a new (analytic) kernel) works for different datasets, activation functions and network architectures, as promised in the future work. In practice, there are other hyperparameters for the network + training that need to be tuned, and the initialisation hyperparameters could be added to this procedure without adding much complexity.

5. clarity: Whilst the exposition of the covariance function and single hidden layer nets is great, I found the presentation for the experiments less organised. It is not clear to me what is the criterion for best and least accurate hypers and why the proposed method seems to be poorer at epoch 0 given that this initialisation is data-dependent (does this mean the marginal likelihood does not correlate with the performance at init?). It’d be clearer if some descriptions of fig 3 and table 1 (and the error bars across multiple runs) are provided.

Overall: I think the proposed method is promising and the topic is of importance. However, the submission in its current form, I think, is not ready for acceptance, primarily due to the lack of evidence of improvement over previous techniques and a more realistic experimental set-up. The additional work listed in 4 and additional experiments/clarifications will greatly strengthen the work.

Minor:

one hot encoding sentence in sec 2.1 is confusing
2.2. first paragraph and third paragraph: guassian -> gaussian


Update 1: I appreciate the authors' response. I think including the "on-going research" points as the authors brought up  and improving the clarity will greatly strengthen the submission. I still think the current form is not ready yet and would like to keep my score as is.
review_title: Promising but experimental set up is too simplistic and there is no clear improvement over simpler approaches

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##########################################################################

Paper Summary: 

The problem this work addresses is the design of a principled way to obtain an appropriate weight initialisation for neural networks. Building on the equivalence between a shallow, infinite width neural network and a Gaussian process with an appropriately chosen kernel function, the authors set off with the key idea of finding the optimal kernel parameters of the GP associated to a neural network, by maximising the marginal log likelihood of the GP, and use such parameters to initialise the weights of the neural network.
A simple experiment on regression over labels of MNIST demonstrates that the approach is sensible.



##########################################################################

Reasons for score: 

Overall, I liked the intuition behind the idea of this work. However, I have several concerns for accepting this paper. First of all, the key to the success of current initialisation methods (e.g. He’s initialisation) lies in their simplicity and low computational costs. The proposed method, instead, requires inverting matrices, that cost $O(n^3)$. The authors acknowledge this fact in their concluding remarks, alluding at the fact that maximising the marginal log likelihood on a very small subset of training points could work, thus substantially reducing computational overheads. Unfortunately, this is not shown in the paper, and needs to be properly addressed.
Second, the analytical derivation of this work applies to shallow models, and by no means it can be applied to modern deep networks without resorting to some sort of model approximation. This is not discussed.
Third, the editorial quality of this paper can enjoy some improvements, especially concerning the mathematical notation which is inconsistent and confusing.


##########################################################################Pros: 

Positive points: 

1) I liked the idea of this work, and I think there is a lot of potential to develop it further. In my opinion, the authors should look at (deep) GP approximations, e.g. through random Fourier features, which are also amenable to approximately describe deep neural network architectures. The key for the success of the idea is: make sure the computational cost is commensurate to the improvements in terms of “performance metrics” with respect to existing (and much cheaper) initialisation methods.

2) The main technical contribution of this work is an alternative derivation of an appropriate covariance/kernel function that is equivalent to ReLU activation functions. The authors should focus on the key advantage of this alternative derivation. From the appendix, it seems to relate to its ability to tackle bias terms more comfortably, but from the experiments (where biases are zero) this does not seem to be sufficiently compelling.

 
##########################################################################

Negative points:

1) As alluded above, my main negative point relates to the computational cost of the proposed method, which is $O(n^3)$, especially when compared to existing methods that cost essentially nothing. As such, the authors should work hard in convincing the reader and the practitioner that there is really a competitive advantage in using the proposed approach. If the method was applicable to larger networks and larger datasets, its cost would be prohibitive. As such, it is important to dig more into GP model approximations, which may be the key for the applicability of the scheme in the general case (see e.g. Cutajar, ICML 17). 
From a more philosophical point of view, however, I have an additional observation. Once you find the optimal parameters for the kernel function of a GP, which in the particular case of this paper is used for regression (hence, analytical solutions for the predictive distribution are available), why not using directly the GP for the problem at hand, instead of “reverting” to a simple neural network? Since the highest price is already payed (inverting a matrix), then it could be questionable to “throw away” the GP and use a neural network.

2) The notation in sec 2.1, 2.2, 2.3, and 2.4 can enjoy some polishing. There are multiple definitions that use the same symbol ($d_{in}=N_0$, $N_1=d_{in}$, $y$ is both a target variable, and a different input than $x$, $f$ is used both to indicate the input/output mapping, the GP prior on $f$, and a probability density function on the weights and biases, $U$ and $V$ have been defined as $z$ elsewhere, …) 

3) The MNIST classification as regression example should be clarified. First of all, despite being a known “trick”, regressing over labels requires some care (see for example Milios, NeurIPS 18). Additionally, my understanding is that the maximisation of the marginal log likelihood is done on a “grid” of kernel parameters, instead of the more general approach using L-BFGS, gradient descent, or conjugate gradient, to name a few. This however, reiterates point 1), that is: you transform parameter initialisation of a neural network into a rather costly optimisation problem. Is it worth it?

 
#########################################################################

Additional comments:

I think this paper has some potential, but it requires more work: 1) to make it more practical, 2) to extend it to neural network architectures adapted to work on more “involved” data, 3) to improve exposition.

review_title: A potentially nice idea, but more work is required

Meta-review: Addressing the initialization issue in DNNs is an important topic, and the proposed approach is found by the reviewers to be interesting. However, the reviewers feel that to clearly promote this research beyond the 'proof of concept' phase, deeper investigation in multi-layer architectures would be required. This would raise the significance of the paper. Besides extending the study to deeper networks, the paper could also benefit from elaborate experiments to increase convincingness, in particular by addressing R4's concerns regarding robustness of performance e.g. on small dataset sizes. Finally, the methodology is sound and the authors clarify the significance of the ReLU associated covariance; however, overall the paper does not offer significant technical advancements that could make up for the shortcomings in the areas discussed above. 
Decision: Reject