Guiding Neural Network Initialization via Marginal Likelihood Maximization
Authors: Anthony Tai
Chunfeng Huang
Venue:  (2021)
Keywords: Neural networks
Gaussian processes
model initialization
marginal likelihood

Abstract: We propose a simple, data-driven approach to help guide hyperparameter selection for neural network initialization. We leverage the relationship between neural network and Gaussian process models having corresponding activation and covariance functions to infer the hyperparameter values desirable for model initialization.  Our experiment shows that marginal likelihood maximization provides recommendations that yield near-optimal prediction performance on MNIST classification task under experiment constraints.  Furthermore, our empirical results indicate consistency in the proposed technique, suggesting that computation cost for the procedure could be significantly reduced with smaller training sets.    

Under review as a conference paper at ICLR 2021
GUIDING NEURAL NETWORK INITIALIZATION VIA MARGINAL LIKELIHOOD MAXIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a simple approach to help guide hyperparameter selection for neural network initialization. We leverage the relationship between neural network and Gaussian process models having corresponding activation and covariance func- tions to infer the hyperparameter values desirable for model initialization. Our experiment shows that marginal likelihood maximization provides recommenda- tions that yield near-optimal prediction performance on MNIST classification task under experiment constraints. Furthermore, our empirical results indicate consis- tency in the proposed technique, suggesting that computation cost for the proce- dure could be significantly reduced with smaller training sets.
INTRODUCTION
Training deep neural networks successfully can be challenging. However, with proper initializa- tion trained models could improve their prediction performance. Various initialization strategies in neural network have been discussed extensively in numerous research works. Glorot and Bengio (2010) focused on linear cases and proposed the normalized initialization scheme (also known as Xavier-initalizztion)) Their derivation was obtained by considering activation variances in the for- ward path and the gradient variance in back propagatiin.. He-initialization (He al., 2015) was developed for very deep networks with rectifier nonlinearities. Their approach imposed a condition on the weight variances to control the variation the imput magnitudes. Because of its success, He-initialization has become the de facto choice for deep ReLU networks. While Glorot- and He- initialization schemes recognize the importance of and make use of the hidden layer widths their formulation, other methods were also suggested to improve training in deep neural networks.
Mishkin and Matas (2016) demonstrated that pre-initialization with orthonormal matrices followed by output variance normalization produces prediction performance comparable to, if not better than, standard techniques. Additionally, [Schoenholz et al. (2017) developed the bound on the network depth based on the principle of 'Edge of Chaos' given particular set of initialization hyperparame- ters. Furthermore, [Hayou er al. (2019) showed that theoretically and in practice proper initialization parameter tuning with appropriate activation function is important model training for improved performance.
Neal (1996) showed that as fully-connected, single-hidden-layer feedforward untrained neural network becomes infinitely wide, Gaussian prior distributions over the network hidden-to-outuut weights and biases converge to a Gaussian process, under the assumption that the parameters are independent. In other words, the untrained infinite neural network and its induced Gaussian process counterpart are equivalent. Also, as result of the central limit theorem, the covariance between network output evaluated at different inputs can be represented activation function. Intuitively, we could therefore relate the prediction performance of an untrained, finite-width, single gagle-hidden-llyyee fully-connected feedforward neural network to Gaussian process model with a covariance function corresponding to the network's 's activation function.
function of the hidden node
In this work propose simple and efficient method that learns from training data to guide the selection of initialization hyperparameters in neural networks. Marginal likelihood is a popular tool for choosing kernel hyperparameters in model selection. Its applications in convolutional Gaussian processes and deep kernel learning are discussed, respectively, in (van der Wilk et al. 2017) Wilson et al. 2016). Our method aims to synergize this powerful functionality of marginal likelihood and

---

Under review as a conference paper at ICLR 2021
the relationship between untrained neural networks and Gaussian process models to make recom- mendations for neural network initialization. We first derive the covariance function corresponding to the activation function of the network whose prediction performance wish to evaluate. We then employ marginal likelihood optimization for the Gaussian process model to learn hyperparameters from data. We hypothesize that the optimal set of hyperparameter values could improve initialization of the neural network.
APPROACH
To assess proposed method, build neural network and Gaussian process model with corresponding activation and covariance functions. With the Gaussian process we estimate the co- variance hyperparameters from training data. These hyperparameter values are then applied in the neural network to evaluate and compare its prediction accuracy among various hyperparameter sets.
We first describe the structure of the neural network, followed by the Gaussian process model and the underlying reason for employing the marginal likelihood. Then, given the network activation function we proceed derive closed form representation of its counterpart covariance function.
2.1 SINGLE-IIDDEN-LAYERR NEURAL NETWORKS
Our neural network model is fully-connected. single-hidden-layer feedforward network with 2000 hidden nodes and rectified linear unit (ReLU) activation function. Following (Lee et al. 2018), conduct empirical study by considering classifying MNIST images regression prediction. Inasmuch as the network is designed for regression, we choose the mean square error (MSE) loss its objective function, along with Adam optimizer, and accuracy the performance metric. In addition, one-hot encoding is utilized to generate class labels, where an incorrectly labeled class is designated 10.1, and correctly labeled class 0.9 For example, the one-hot representation of the integer 7 is given by [-0.1, -0.1, 10 -0.1, -0.1, 10.1, -0.1, 0.1,0.00. 0.9, -0.1,-0.1]
Input Layer
1 Hiddon Layer
Output Layer
1dt node
1.0
(x)
+
xx(00
j node
Regression Output
+ (r)
0.4
N nude
0.2
+0,00
(x)
Figure I: A single-hidden-layer, fully-connected feedforward neural network for regression predic- tion. Left panel: Structural diagram of the neural network. Right panel: ReLU activation function: Φ(a) = (a)+ = max(0, a) = a for a 2 0; ((a) 0 otherwise.
As shown left panel of Figure I the single-hioden-layer neural network has a set of inputs denoted by [ & 21, dm with input layer width d, No 28x28 = 784. The model's weight and bias parameters from 4th input node to jth hidden node are W0 of No b. iid A' of), and W by. Similarly, the weight and bias parameters from jth hid- nid den node to tth output node with hidden layer width N1 = dm = 2000 are W N/(0, N ). ~ (O of),, and W! JL b/. For regression models the output layer has single node, and therefore i (1). The ReLU nonlinearity is depicted in the right panel of Figure (1)
(O,
2

---

Under review as a conference paper at ICLR 2021
The imput to each hidden node nonlinearity (the pre-activation) is represented by 29(x) bg + Σ W =k. while the hidden unit output after the nonlinearity (the post-activation) is denoted by =)(x) (() (x)), 2 N. Since typically apply linear activation function in the
output stage of a regression model, the model output is simply 2 (I) = b! + Σز= W (z))
2.2 GAUSSIAN PROCESSES
A Gaussian process Mac 1998; Neal 1998 Williams and Rasmussen 2006 Bishop 2006) is
set of random variables any finite collection of which follows multivariate normal distribution. A Guassian process prediction model exploits this unique property and offers a Bayesian approach to solving machine learning problems. The model is completely specified by its mean function and covariance function.
By choosing particular covariance function, prior distribution over functions induced which, together with observed inputs and targets, can be used to generate prediction distribution for making predictions and uncertainty measures on unknown test points. These capabilities allow Gaussian processes to be used effectively in many important machine learning applications such as human pose inference (Urtasun and Darrell 2008 and object classification (Kapoor et al.] (0100). Recent rescarch works also apply Gaussian processes in deep structures for image classification (van der Wilk et al. 2017) and regression tasks (Wilson et al. (016).
To help achieve optimal performance for Guassian process prediction select suitable covariance function and tune the model by adjusting hyperparameters characterizing the covariance function. This can be accomplished by applying the marginal likelihood which is crucial feature that enables Gaussian processes to learn proper hyperparameter values from training data.
2.3 HYPERPARAMETERS AND MARGINAL LIKELIHOOD OPTIMIZATION
We briefly describe the procedure for estimating optimal hyperparamter values via maximizing the Gaussian process marginal likelihood function.
Consider a of N multidimensional input data X { x; RDD and target {u:) Y/ R. For each input z; have corresponding input-output pair (I,, ys), where the observed output target is given by yi = f(;)) + c,, with data noise & ~ N00 o.). We model the input-output latent function
Gaussian process:
f(x,) ~ ()(()))),)) Mx,,);))
where we customarily set the mean function u(x;)= = E[f(x;)] = 0, and denote k(I;, 2,) as the covariance function.
The marginal likelihood (or evidence) (Williams and Rasmussen, 2006 Bishop 2006) measures probability of observed targets given input data and can be expressed the integral of the product of likelihood and the prior, marginalized over the latent function f:
X(vlX)= = f plu, fIX) = -f DiuI,, XxlUl1x))((IX))
(1)
The marginal likelihood can be obtained by either evaluating the integral 1 or by noticing {us ===1 = +e, which gives us y|X ~ (++?? where K. = [k( 2y)l and N by N covariance matrix and identify matrix, respectively. As result,
X)y N/21) + 121/22 exp (-""((x
To facilitate computation, evaluate the log marginal likelihood which is given by log x(y]X) = T o
N
log IK + 03,1 2
log 2n.
(2)
We are reminded here that the marginal likelihood is applied directly on the entire training dataset, rather than a validation subset. In addition, Cholesky decomposition (Neal 1998) can be employed to calculate the term (K. + r) in equation

---

Under review as a conference paper at ICLR 2021
2.4 RELU COVARIANCE FUNCTION
With the structure of the single-hidden-layer ReLU neural network defined, we proceed to study its corresponding ReLU Gaussian process.
The ReLU covariance function is developed to estimate the covariance at the output of the ReLU neural network model. Our alternative derivation was inspired by the work on arc-cosine family of kernels developed in (Cho and Saul 2009). In our work we first derive the expectation of the product of post-activations, instead of on the imput to the nonlinearity (Lee al.) 2018). Then, we apply the output layer activation function on the post-activation expected value. It can be shown that
the resulting representations are equivalent. The complete derivation of our expression is provided in the Appendix 5
Referring to Figure we consider input vectors x=, y Rdii. The initial weight value is drawn
randomly from the Gaussian distribution fwy, (0, (x) and bias value from fog (N of)-
The expected value of the product of post-activations at the output of the jth hidden node is com- puted as
E|X (x X (2))
max (b) + ww x²) max + wg. y2) fog wg (b, w) duf ab}
(6) + w x (b) + w/ x0)+ Sog (b, (b, w) dw3 db
(3)
Suppose we denote the pre-activations as
dum
U= b9 + WD x0 = =bg +
Wy. jk ~ N00, +1
k=
V = b9 + W 100 = 69 +
w ju. of +
It can be shown that the random variables U, V have a joint Gaussian distribution:
o + af ||||
o y)
(10 E), where Σ
V
oz
o (x y)
+ 1y
for simplicity we let x 20, y = y.
We can therefore write expression
1
exp (-
2 (M, (M, w) (u, ())) du dv.
1
MU
||
Now let D == [
(o}
> (o} + (1v1?) - (oî + oz( x 4))² and
E-1_
A11
012
where an 1 11)111²), a22 1/1 (0
100 + o |00)))
021
-1
(oî + o v)).
412 = 0g1
D
With polar coordinate transformation: u Jan cos a, U = vays sin a, expression
can be
further reduced to
2 sin 2a da
4x 122 a11 022 Jam (1- cos db sin 2a
2x sin's (sin(1)) + (IT - ②)) cos(
where o = coos- (1
(anna
4

---

Under review as a conference paper at ICLR 2021
With some algebraic operations and after computing the entries in E-!, arrive (x) (y)
o + |||||| ||:1|
|1u11300]] +0/+11u1
(sin Φ + (T Φ) cos s)
2π
of + (x y)o
where o = cos-
[0} |
1/2 (oB + 11u11302
1/2
To compute the expected value, E|X, (x)] I max(b (b + f1gg w., (b, w) dw db, we denote
b N( of ) and apply the change in variable 1
20 20z2² u²
t, where o²
of + -=|ا: to obtain
[x) f (u)+fo
20z u²
du
V2TO
0² dt
e
V2TO
22
0f + oz |0| V2T
The covariance function at the network output is therefore determined to be
N1
Ni
N1
N1
E
(b.
+ Σ (z))
+
W.lxx()))] - E
Σ
w Xy
(z)
6 +
Σ
w.lxa())
==1
E((b)) + Σ B(WW))9000
2n / VoB VoB + - 1 12 of + 1
of oh +
2
N₁ E
2n oo
+
2 |||| Vot of 2
2 +
a²
of + |k=]3/02]
(o3 1lu13=2) & sin o + の)
cos Φ 1).
2x
2.5 GAUSSIAN PROCESS PREDICTION: A SIMULATION
Performing simulations allows us to explore and understand some properties of the models we wish to study. Simulation results also offer the opportunity for evaluating model precision and insight into observed events.
To demonstrate making predictions with Gaussian process regression model, we borrow equations from (Williams and Rasmussen] (2066) where the formulation of Gaussian process predictive distri- bution treated in great detail.
Given the design matrix X = J. Iz E RD, observed targets y = ( E R, unknown test data X., and their function values f. == f(X.), the joint distribution of the target and function values is computed as
J N
K(X. X) o2 K(X. X)
x. x(X.,x.)) ))
where K (XX, XX) represents the covariance matrix of all pairs of training points, K(X,X, denotes that pairs of training and test points, and K(X,,,, X.) gives the covariance matrix of pairs of test points.
5

---

Under review as a conference paper at ICLR 2021
The prediction distribution is the conditional distribution y, ~ NN (u,, Σ.)
with mean function μs. ((X,, x)) and covariance E.
The simulation starts out with setting the hyperparameters of the ReLU covariance function to (3.6, 0.02). chosen from o E [0.4, 1.2, 2.0, 2.8, 3.6). and of [0.0001, 0.01, 0.02]. We randomly select set of 70 training and 30 test location points from 100 values evenly spaced in the interval [0.0, 1.0]. Ten sample paths, as shown in the top left panel of Figure 2 are generated from the design Gaussian process model. Their sample mean produces 70 training target and 30 test values. We then estimate the optimal hyperparameters from the training targets via evaluating the marginal likelihood, equation P over the design ranges of og and og.
The maximum marginal likelihood is obtained at (# af] = [3.6, 0.02) which is the design hy- perparameter pair. The minimum marginal likelihood is obtained at {ō o {3.6, 0.0001 A Gaussian process model is then built with the optimal hyperparameter pair to make predictions for the 30 test location points, The model accuracy is assessed with RMSE of 0.00051. Additionally we overlay the predicted and true test target values, as shown in the top middle panel Figure 2}. to detect any prediction errors. We plot the line of equality further validate the estimated nyperparameters, as depicted in the top right panel of the figure.
The evaluation process is repeated applying the hyperparameter pair {ō?, o8) which produces prediction RMSE 0.00188, over 3 times as large as the optimal case. The accuracy plots shown the bottom panels of Figure (2) indicate some prediction errors.
0.25
0.25
0.40
Target
0.20
Une of Emiunity
Desponee value
0.20
Rerponse value
0.15
Pradctioon
0.15
0.00
0.10
0.20
0.00
1.5
0.05
0:15
0.70
location
Test location
True Truc Values
0.25
Prediction Mean Target
Une of Setwary
0.20
Hespuree velue
0.15
0.10
0.05
0.05
0.80
0.0
0.5
1.0
3.5
0.00
0.05
0.10 0.15
0.20
0.25
1est location
True Values
Figure 2: Gaussian process regression prediction on simulated data. Top left: 10 sample paths gen- erated from Gaussian process model with hyperparameters (o Point-wise visual comparison between predicted and true target values for the optimal hyperpa- rameter pair (3.6, 0.02), showing good prediction results. Top right: The line of equality further confirming the prediction accuracy. Bottom left: Point-wise visual comparison for hyperparameter pair (3.6, 0.0001). Bottom right: Prediction errors revealed with the line of equality.
(a2. of) = (3.6. 0.02). Top middle:
Our simulation results agree with the principle that through optimizing the marginal likelihood of the Gaussian process model, we could estimate from training data the hyperparameter values most appropriate for its chosen covariance function.

---

Under review as a conference paper at ICLR 2021
3 MNIST CLASSIFICATION EXPERIMENT
We conduct classification experiment the MNIST handwritten digit dataset LeCun 1998) making use of corresponding ReLU neural network and Gaussian process models. As in (ee et al. 2018), the classification task on the class labels is treated Gaussian process regression (also known as kriging in spatial statistics (Cressie 1993)).
It is necessary to point that the goal of this work is examine using the marginal likelihood to estimate the best available initial hyperparameter setting for neural networks, rather than determin- ing the networks' optimal structure.
Our experiment consists of three main steps: (A) searching within a given grid of hyperparameter values for the pair {o [5. that maximizes the log marginal likelihood function of the Gaussian process model, (B) evaluating prediction accuracy of the corresponding neural network at each grid point {o o including {o ]. and (C) assessing neural network performance over all tested hyperparameter pairs.
3.1 PROCEDURE
The workflow for the experiment is as follows: we set up grid map of E (0.4, 1.2, 2.0, 2.8, 3.6), of E {0.0, 1.0, 2.0}. Then, N samples are randomly selected from the MNIST training set to form a training subset, where N is the training size. This is followed by computing the log marginal like- lihood (equation 2) each grid point. This allows us to identify the hyperparameter pair {ō) ) that yields the maximum log marginal likelihood value.
On the neural network side, we build fully-connected feedforward neural network with single hid- den layer width, hidden. width, of 2000 nodes, Adam optimizer, and mse loss function. Since the net- work model is fully connected, the size of the input layer d,, is 28(pixels) 28(pixels) 784. Prior to training, the initialization parameters {w, b) are set by sampling the distributions N/ 2 /dn) and (O, of) for weights and biases from the input to the hidden layer, and (O /2000) and (OO of) for weights and biases from the hidden to the output layer. The neural network is then trained with the training subset generated previously. We compute the model classification accuracy on the MNIST test set and repeat the procedure over the entire grid map of hyperparameter pairs.
To investigate the usefulness of our proposed approach for assisting model initialization, we employ He-initialization approach as benchmark to measure numerically and graphically our neural net- work performance over all tested hyperparameter pairs. Additionally, we check for recommendation consistency.
3.2
RESULTS
Applying the method described Section 2.3 for estimating model hyperparameter pair we obtain a consistent recommendation of (a of) (3.6, 0.0).
1.00
1.00
3.00
0.90
0.90
Trening AcTuncy
0.80
Traniing ACCUSIY
0.80
Traniing Accunecy
0.80
0.70
0.70
0.60
0.50
0.50
Most Accurate
0.50
Accunate
Least Accurate
Least Actunate
Heint
0.40
He Wift
Heinit
Curs
Ours
Ours
0.30
12.5 0.30
0.30
0.0
2.5
5.0
100
12.5
Traniigg Epoch
Training Spoch
Training Esech
Figure 3: Comparing MNIST training accuracy over various training sizes. We observe that the convergence rate based increases. This suggests that our technique may potentially be efficient for guiding deep neural network initialization. Left: train size==0000 Middle: train wain-size=3000 Right: train size=5000.
method approaches that using He-initialization as the training size

---

Under review as a conference paper at ICLR 2021
After running 250 training epochs, convergence of the neural network model and its prediction accuracy are studied for different training sizes. We observe that training based on our initialization approach converges to that based He-initialization the size of training samples increases, as shown in Figure 3 This seems to suggest that our approach may be used as an efficient tool for recommending initialization in deep learning.
worth noting that the Gaussian process model marginal likelihood consistently suggests the hyperparameter pair (o (o o (3.6, 0). The fact that the bias variance o is estimated be coincides with the assumption that bias vector being O in (He et al. 2013).
Table lists neural network model prediction accuracy based on, respectively, approach and He-initialization scheme, against the best and the worst performers. The results indicate that more frequently our approach achieves slightly better accuracy than based on Hc-initialization. However, neither approach reliably gives an estimate of weight variance close to that for the best case.
Table I: Single-hidden-laye layer fully-connected neural network model prediction accuracy on MNIST test set, and associated hyperparameter pair.
Best Case
Worst Case
He-Init
Ours
Size
Acc.
(o? of)
Acc.
(o of?
Acc.
(o of)
Acc.
(a 007 o.)
10000
96.85
(2, 0)
96.04
(0.4, 2)
96.85
(2, 0)
96.60
(3.6, 0)
20000
97.25
(2.8, 0)
96.70
(3.6, 1)
97.01
(2.
97.09
(3.6,
30000
97.50
(1.2,
96.91
(2
97.07
(2
97.29
(3.6, 0)
40000
97.43
(0.4, 0)
97.16
(0.4, 2)
97.35
(2
97.42
(3.6,
50000
97.71
(3.6,0)
97.29
(0.4, 2)
97.50
(2,
97.71
(3.6, 0)
4 DISCUSSION AND FUTURE WORK
In this work we propose simple, consistent, and time-efficient method to guide the selection of initial hyperparameters for neural networks. We show that through maximizing the log marginal likelihood we can learn from training data hyperparameter setting that leads to accurate and efficient initialization in neural networks.
We develop an alternative representation of the ReLU covariance function to estimate the covariance the output of the ReLU neural network model. We first derive the expectation of the product of post-activations. Then, we apply the output layer activation function on the post-activation expected value generate the output covariance function. Utilizing marginal likelihood optimization with the derived ReLU covariance function we perform a simulation to demonstrate the effectiveness of Gaussian process regression.
We train a fully-connected hagle-hidden-layerr neural network model to perform classification (treated as regression) on MNIST data set. The empirical results indicate that applying the rec- ommended hyperparameter setting for initialization the neural network model performs well, with He-initialization scheme as the benchmark method.
A further examination of the results reveals consistency of the process. This implies that smaller training subsets could be used provide reasonable recommendation for neural network initializa- tion on sizable training data sets, reducing the computation time which is otherwise required for inverting considerably large covariance matrices.
The main goal of our future research is to investigate if our proposed method is adequate for deep neural networks with complicated data sets. We wish to ascertain consistent recommendation could be attained by learning from larger data sets of color images via marginal likelihood maxi- mization. We will attempt to derive or approximate multilayer covariance functions corresponding to various activation functions. Deep fully-connected neural network models will be built to perform classification on CIFAR- 10 data set. Our hypothesis is that learning directly from training data helps to improve neural network initialization strategy.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Christopher Bishop. Pattern recognition and machine learning. 2006.
Youngmin Cho and Lawrence K. Saul. Kernel Methods for Deep Learning. In Y. Bengio, D. Schu- urmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 342-350, Curran Associates, Inc., 2009. URL papers.......... learn learning..po
nips. cipp.cc//aperr33// -erd-28-kereeee kernel forthodss-oo--eeee
Noel AC Cressie. Statistics for spatial data. John Willy and Sons. Inc., New York, 1993.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. AISTATS, page 2010.
Soufiane Hayou, Amaud Doucet, and Judith Rousseau. On the Impact of the Activation Function on Deep Neural Networks Training. JCML, May 2019. URL 06853
XrV::: 1902.06853.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing [cs], February 2015.
Human-Level Performance on ImageNet Classification.
URL http:h/aaxxi.ccooo
abs/1502. 01852] arXiv: 1502.01852.
Ashish Kapoor, Kristen Grauman, Raquel Urtasun, and Trevor Darrell. Gaussian Processes for
Object Categorization. International Journal of Computer Vision, 88(2):669-188 June 2010.
ISSN 1573-1405. doi: 10.
URL
$11263-009-0268-2
Yann LeCun. THE MNIST DATABASE of handwritten digits, 1998.
URL http::/yaat
lecun. com exdb mnist/
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha
Sohl-Dickstein. Deep Neural Networks as Gaussian Processes. 1XXiv:7711.0066: [cs, stat), arXiv: 1711.00165.
March 2018. URL/htt:: http:p://aaxiv.eeee http://aaxx.vv.o////////// 00165
David MacKay. Introduction to Gaussian processes. Citeseer, 1998.
Dmytro Mishkin and Jiri Matas. All you need is good init. ICLR, February 2016. URL http: arXiv: 1511.06422
Radford M. Neal. Bayesian Learning for Neural Networks, volume 118 of Lecture Notes in Statis-
fics. Springer New York, New York, NY, 1996. ISBN 978-0-387-947244 978- -412--4451 URL attp:://iie springer com 10077
0. doi: 10110077978-1146222774444 41144612--755-
1-4612-0745-0.
Radford M. Neal. Regression and classification using Gaussian process priors. Bayesian statistics, 6:475, 1998.
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein, Deep Information Propagation. ICLR, April 2017. URL bttp://aaxxivooog/aa//l11...... 1611.0.
arXiv:
Raquel Urtasun and Trevor Darrell. Sparse probabilistic regression for activity-independent human pose inference. CVPR, 2008
Mark van der Wilk, Carl Edward Rasmussen, and James Hensman. Convolutional Gaus- sian Processes.
I. Guyon, V. Luxburg. Bengio, H. Wallach, R. Fergus, Vish- wanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30. pages 2849-2858. Curran Associates, Inc., 2017. http:htttp//papr..... nips cctppperrr
processsss.....
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, 2006.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric Xing. Deep kernel learning. In Artificial Intelligence and Statistics, pages 370-378, 2016.

---

5 Under review as a conference paper at ICLR 2021
5 APPENDIX
Covariance Function the Output of ReLU Neural Network
Our derivation follows the work on arc-cosine family of kernels developed in (Cho and Saul, 2009). However, instead of applying coplanar vector rotation in calculating the kernel integral, we recognize that the integrand can be written in terms of two jointly normal random variables. This helps facilitate the computation which becomes more involved when both the weight and bias parameters are included.
The derivation is also made to conform to the arc-cosine kernel by utilizing the identities Saul 2009 equation (17), (18)) to give us
Cho and
Φ dnj sin Φ
7=0 cos Φ COs v)
sin 20
d0
00= cos Φ sin 20)
sin(o) (Ø) + (π - の) cos(
(4)
sin
Equation (4) derived, with the substitution = 2(0 7 ) as follow:
sin 20
d0
0= (1 sin 20)*
2 cos
2 dn
n= (1
cos á cos 17)
0
1
dn
cos
m cos & cos T
0
-1 θ
cos
6 sin Φ sin( 0o
sin Φ
1
sin( + (π - Φ) cos(Ø)
sin
Denote the input layer (layer 0) weight and bias parameters as b9 ~ N' of) and w lid where W9 for all &s {I,... din).jj E {L...., NN...
o N(O
di..
The expected value of the product of post-activations at the output of the jth hidden node is com- puted as
E|X
(3))
max(b) +wg. 2²) max(b) w y yf) w wp(b, w) duf ab}
(b) w/J x²)+ (b) ww y Sog Seg. wb (b, w) duo db
(5)
Each pre-activation can be written in terms of a random variable:
U U= b + WD x0 = 69 +
whrh ~ N/ + 12
V 69 + WT n 69 +
W y/ v& ~ +2
10

---

Under review as a conference paper at ICLR 2021
Since E[U] E[V] 0, their covariance can be expressed
cov(U, V) El(b) + W3 x²) (b) + W.. )
din
dim
E((0)) E
W W jk
Rim
='=1
dip
din
oB
E W 1
din
= 0h + 0
0 o (For simplicity we set = =", 5r)
k=1
= of + σ (x. (v)y)
This implies that the random variables U, V have a joint Gaussian distribution:
y v
o 1=
t y)
E), where E
(x
of + +
We can, therefore, rewrite equation 5 as
11.
2T b exp ( - (u, - vu) du de
(6)
Denote D :==1a (oî + 1 (oī + +
(of + (x y))² and
Σー
a1
012
a21
022
with a11
D (o + (
= D
(oū + +z and
[o + o ())
a1 = 021 = D
We therefore have:
D21 - - = D
D (0 + (3lllll))) (oī + - - - (0) - -z 3))²
)
(of + ( + )lll) (o) + a (x y)) 2)
1. (by definition)
The exponential term in equation
66) then becomes:
1
1 (anu²² + 2a + u22v?).
2 (u, x) (u, T)T
We now make use of the transformation from Cartesian to polar coordinates by setting
COS U
sin
U11
Va2z
an = 2200 a, az2
san
The Jacobian J is calculated as
Ouu, v)
r
0( a)
Va12022
11

---

Under review as a conference paper at ICLR 2021
Equation 6 can in turn be expressed
1
00
2 sin 20x
- 1r"cos +
2012t² sin a cos ar
r dr da Van1as
exp
+
2xD1/2
a= J Jr=
2/11100 2/22022 022
V011022
1
x0
(p²
11 sin 2a
sin 2a do
eep
[1 +
dr /Q1102
(8)
4T'D1/211022
Ja=0
r=
2
01 sin 2a
Next, we need to show that H == 1 +
0 to ensure the expression in ( is bounded.
vai1022
First, since | -- |1 + ||yl|
2(x y)
0
| + |lvll2 2 2(x y). and let the angle
between the vectors x, y be 0 cos-
x
we have
| |1yll
o + ( v) 2
0 2 ||01 3)
oo + 13ll1
y) + 0 (08)
+ of + (0 v) 20
(z 20
|2 [1vl3) of 22(z y)
of (o( [yll)
1.
This means that we can define a quantity & as
(o) + a (x w)
Φ
cOs
(of + 211 (o) + + 1/2
D (of o (x v)
Co8
(o] +
o | |33 + +
D
cos -1
112
Va11a22
-212
CO8 Ø =
Va11022
This also leads to
11 sin 2a
H = 1+
/a11022
-1
(o + oz v)) sin 2a
=1+
(og + +0 +2.ll111))))))
D
(of + oz v) sin 2a
=1
(0f + 1 + ( 1/2
(0 + 100 v))
(of + +8 1/2
20
12

---

Under review as a conference paper at ICLR 2021
With change of variables, now evaluate the integral involving the parameter in expression as follows.
r²
2n
2 2\
Let 17
H. Then r
dr=
=
"
1/2 dr. We have
2
H
H
H
2
11
2V
H
2
-
dn
1100
2
1/C
dn
H22
1==0
2
116 11
dn
H2
1-=0
2
2
[(2)
H2
H2
2
(1 + 01 sin 20
/a11022
2
(from equation 9
2-
1 cos & sin 20x
The complete expression
g
becomes
sin 2a
2 da
4 D1/2
22//211110 a22
(1-cos Φ sin 2ax
(sn(()) + (x1 - Φ) (0)cs)))
(from equation 4
2n 11/ /a///aa111 a22 sin
-012
where o- = cos
Vai1a22
Finally,
2T DT 1/2 "11 1111122222 sin3d
= 2=''22211122
011222 (1
2T
(1
3/2
112
-1/2
3/2
2/ (a11023)
aa11a12
013)
11/
3/2
2π
D Daaai1a22
a1z)
-1/2
3/2
(from equation F
2t (of ||=)|| (o) + o (lllll)))
The expected value of the product of post-activations at the output of the jth hidden node in the first hidden layer is therefore determined to be
E (x))
+o 13) (of + oz (wllll))) 1/2
(oī
(sin()) + (TT Φ) cos(e)))
2T
(0% + (x v))
where @ cos
(10) + 02 (3) + o 11111113))
1/2
13

---

Under review as a conference paper at ICLR 2021
To compute the expected value, (x) f max + fo) (b,w) dw db, denote
U b+ w N(0, of (0=|==1))) and apply the change in variables:
202 u² = t, where o²
of + |=1| to obtain
[x)] =
fu)+fu (u)du
zoou u2
11
e
du
V2TO
di
P.
V2TT
V2k
Vol + o |
V2T
The covariance function at the network output is therefore determined to be
N W xx))
N
N1
N₁
E
(0.
+
+
- B [
(z)
b +
Σ
(w))
j=1
E(())) +
1/2 VoB 2 - 12
of + 11yl
2x
of +
Le N₁ F
X, (v))
1
Vož + of + 22111112
N1
2T
100 1/2 + +lul8822) $ (sin + (क Φ) cos Φ 1).
2 +
10 +
||
2x
14


Reviews:
[{"review_id": "c64ID2O5xWE", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper considers the initialisation of fully connected neural networks with ReLUs.\nBy using the GP-one hidden layer neural network equivalence result, the paper considers using the equivalent GP and its marginal likelihood to decide the initial sampling parameters for the weights and biases. The paper first re-derives the covariance function of the GP corresponding to a single-hidden-layer ReLU network. To motivate the use of the marginal likelihood, the paper provides a simulation showing optimising the marginal likelihood on the data drawn from a GP arrives at hyperparameters that closely match the hypers used to generate the data. An experiment with neural network regression on the MNIST dataset shows the initialisation based on the GP marginal likelihood is promising.\n\nAssessment:\n\nStrengths:\n\n1. The submission tackles an important problem in neural networks -- initialisation as despite a suite of techniques proposed for this task, any new technique that can consistently improve over the old ones in practice or provide more theoretical insights will be beneficial to the field. \n\n2. The topic of connecting (deep) neural networks and GPs and how to exploit this to improve neural network training is important. This submission shows this connection is useful for initialisation, as the marginal likelihood for GP regression can be computed in closed-form and for small dataset, optimising this objective is not costly.\n\nWeaknesses:\n\n3. novelty: The elements of the proposed approach are not novel. The covariance function for ReLU networks is not new and I don’t see the new insights from the new derivation compared to the work of Lee et al (for example: does this make things simpler when moving to deeper networks or a different activation function). The simulation study is useful, but not surprising since given enough data points the maximisers of the marginal likelihood should be the hyperparameters used to generate the data. \n\n4. practical significance: The experiment on the MNIST dataset is promising, however, it fell short of bringing the strengths (1 and 2) home. For example, it is not clear to me why a practitioner should choose the proposed approach which requires training (potentially expensive if the subset is large) over simpler approaches like He-init --- figure 3 shows no improvement over He-init and the accuracy results in table 1 doesn’t really show a clear difference between methods at the end. I’m not entirely sure I got the reasoning of the experimental set-up, e.g. why the training sets are subsets of the full MNIST, and what is the difference in terms of results between different training regimes? The task considered here is MNIST regression with a network with only one hidden layer. It is therefore very hard to justify the ‘near-optimal’ performance claimed early in the paper. It’d be good to show that the same approach (with a new (analytic) kernel) works for different datasets, activation functions and network architectures, as promised in the future work. In practice, there are other hyperparameters for the network + training that need to be tuned, and the initialisation hyperparameters could be added to this procedure without adding much complexity.\n\n5. clarity: Whilst the exposition of the covariance function and single hidden layer nets is great, I found the presentation for the experiments less organised. It is not clear to me what is the criterion for best and least accurate hypers and why the proposed method seems to be poorer at epoch 0 given that this initialisation is data-dependent (does this mean the marginal likelihood does not correlate with the performance at init?). It’d be clearer if some descriptions of fig 3 and table 1 (and the error bars across multiple runs) are provided.\n\nOverall: I think the proposed method is promising and the topic is of importance. However, the submission in its current form, I think, is not ready for acceptance, primarily due to the lack of evidence of improvement over previous techniques and a more realistic experimental set-up. The additional work listed in 4 and additional experiments/clarifications will greatly strengthen the work.\n\nMinor:\n\none hot encoding sentence in sec 2.1 is confusing\n2.2. first paragraph and third paragraph: guassian -> gaussian\n\n\nUpdate 1: I appreciate the authors' response. I think including the \"on-going research\" points as the authors brought up  and improving the clarity will greatly strengthen the submission. I still think the current form is not ready yet and would like to keep my score as is.", "extra_scores": {}, "extra_text": {}}, {"review_id": "X07ykCltA5z", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nPaper Summary: \n\nThe problem this work addresses is the design of a principled way to obtain an appropriate weight initialisation for neural networks. Building on the equivalence between a shallow, infinite width neural network and a Gaussian process with an appropriately chosen kernel function, the authors set off with the key idea of finding the optimal kernel parameters of the GP associated to a neural network, by maximising the marginal log likelihood of the GP, and use such parameters to initialise the weights of the neural network.\nA simple experiment on regression over labels of MNIST demonstrates that the approach is sensible.\n\n\n\n##########################################################################\n\nReasons for score: \n\nOverall, I liked the intuition behind the idea of this work. However, I have several concerns for accepting this paper. First of all, the key to the success of current initialisation methods (e.g. He’s initialisation) lies in their simplicity and low computational costs. The proposed method, instead, requires inverting matrices, that cost $O(n^3)$. The authors acknowledge this fact in their concluding remarks, alluding at the fact that maximising the marginal log likelihood on a very small subset of training points could work, thus substantially reducing computational overheads. Unfortunately, this is not shown in the paper, and needs to be properly addressed.\nSecond, the analytical derivation of this work applies to shallow models, and by no means it can be applied to modern deep networks without resorting to some sort of model approximation. This is not discussed.\nThird, the editorial quality of this paper can enjoy some improvements, especially concerning the mathematical notation which is inconsistent and confusing.\n\n\n##########################################################################Pros: \n\nPositive points: \n\n1) I liked the idea of this work, and I think there is a lot of potential to develop it further. In my opinion, the authors should look at (deep) GP approximations, e.g. through random Fourier features, which are also amenable to approximately describe deep neural network architectures. The key for the success of the idea is: make sure the computational cost is commensurate to the improvements in terms of “performance metrics” with respect to existing (and much cheaper) initialisation methods.\n\n2) The main technical contribution of this work is an alternative derivation of an appropriate covariance/kernel function that is equivalent to ReLU activation functions. The authors should focus on the key advantage of this alternative derivation. From the appendix, it seems to relate to its ability to tackle bias terms more comfortably, but from the experiments (where biases are zero) this does not seem to be sufficiently compelling.\n\n \n##########################################################################\n\nNegative points:\n\n1) As alluded above, my main negative point relates to the computational cost of the proposed method, which is $O(n^3)$, especially when compared to existing methods that cost essentially nothing. As such, the authors should work hard in convincing the reader and the practitioner that there is really a competitive advantage in using the proposed approach. If the method was applicable to larger networks and larger datasets, its cost would be prohibitive. As such, it is important to dig more into GP model approximations, which may be the key for the applicability of the scheme in the general case (see e.g. Cutajar, ICML 17). \nFrom a more philosophical point of view, however, I have an additional observation. Once you find the optimal parameters for the kernel function of a GP, which in the particular case of this paper is used for regression (hence, analytical solutions for the predictive distribution are available), why not using directly the GP for the problem at hand, instead of “reverting” to a simple neural network? Since the highest price is already payed (inverting a matrix), then it could be questionable to “throw away” the GP and use a neural network.\n\n2) The notation in sec 2.1, 2.2, 2.3, and 2.4 can enjoy some polishing. There are multiple definitions that use the same symbol ($d_{in}=N_0$, $N_1=d_{in}$, $y$ is both a target variable, and a different input than $x$, $f$ is used both to indicate the input/output mapping, the GP prior on $f$, and a probability density function on the weights and biases, $U$ and $V$ have been defined as $z$ elsewhere, …) \n\n3) The MNIST classification as regression example should be clarified. First of all, despite being a known “trick”, regressing over labels requires some care (see for example Milios, NeurIPS 18). Additionally, my understanding is that the maximisation of the marginal log likelihood is done on a “grid” of kernel parameters, instead of the more general approach using L-BFGS, gradient descent, or conjugate gradient, to name a few. This however, reiterates point 1), that is: you transform parameter initialisation of a neural network into a rather costly optimisation problem. Is it worth it?\n\n \n#########################################################################\n\nAdditional comments:\n\nI think this paper has some potential, but it requires more work: 1) to make it more practical, 2) to extend it to neural network architectures adapted to work on more “involved” data, 3) to improve exposition.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WtwaY2lQ26h", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThe authors propose a rule for neural network (NN) initialization, which takes into account input data. \n\nThey suppose that weights and biases of a NN are randomly drawn resp. from $\\mathcal{N}(0, \\sigma_w^2 / N)$ and $\\mathcal{N}(0, \\sigma_b^2)$, where $N$ is the number of inputs. Then, they are able to compute *explicitly* the covariance matrix of the corresponding Gaussian Process. Since an explicit result is needed, they concentrate on one-layer NNs.\n\nThey use their explicit formula for the covariance matrix to compute the likelihood of the data, given $\\sigma_w^2$ and $\\sigma_b^2$. Therefore, they are able to select the best pair $(\\sigma_w^2, \\sigma_b^2)$ according to data, i.e., maximizing the likelihood.\n\n### Clarity\nI did not understand the experimental setup presented in Section 2.5. For instance, the train/test location are supposed to lie in the interval $[0, 1]$, but the test location points lie in $[0, 2]$ in the graphs.\nBesides, all the computation of the covariance should be put in appendix.\n\n### Significance\nSince the paper relies on an explicit computation in one-layer NNs, the presented method has a very low significance. At least, the authors should propose an application in deeper NNs, even by making strong approximations. As such, no clue about any generalization is provided.\n\nEdit:\n### Rebuttal\nI did read the authors' rebuttal, and the main issue, i.e. the significance, has not been addressed. I cannot take into account the new experiments, since they are not in the paper. Anyway, an experiment with a 2-layer network would be a significant modification of the present paper, which would be not acceptable during the rebuttal phase.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GFbKYCsLQ5K", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### UPDATE:\n\nPer my response in the thread, I appreciate the authors' replies and updates, but I am keeping my score because\n- Even with the new 2-layer results, I find this experimental setting still too limited;\n- Even within the conducted experimental setting, the benefit of likelihood-guided initialization over He init is not robust (notably on small dataset sizes, which is contrary to the expectation that a good prior will be more beneficial in such cases).\n\n### SUMMARY: \nThe paper proposes to use the marginal likelihood of the Gaussian process (GP) corresponding to the infinite width Bayesian neural network (NN) to select hyper-parameters of the finite width, SGD-trained NN (instead of evaluating performance on the validation set). \n\nExperimentally, weight and bias variance in 1-hidden layer ReLU network are considered on MNIST, and on multiple training set sizes GP-likelihood-selected configuration slightly outperforms the default He-initialization, and slightly under-performs the best possible configurations. \n\nThe paper also provides an alternative derivation of the ReLU kernel.\n\n\n### REVIEW SUMMARY:\nWhile I appreciate and support the question (**Q: Can we use NN-GP likelihood to select hyper-parameters of a finite SGD-trained NN?**) the paper raises, I believe the paper does not answer it (due to very limited experimental evidence), and raising the question itself is not sufficient for publication.\n\n\n### PROS:\n1. I believe the proposed idea is promising. \n2. Answering the question **Q** is certainly of interest to the research community. Unlike training and evaluating on a validation set, marginal likelihood of the NN-GP can be simply evaluated in closed form on a small subset of the training set, potentially making it a cheap proxy for validation performance.\n2. The paper is clearly written and easy to follow.\n\n\n### CONS:\n1. My key concern: unfortunately experimental validation of the hypothesis is not adequate. After reading the paper I remain in the dark regarding whether I should use NN-GP likelihood for hyper-parameter search or not. Precisely:\n\n\t1. The idea is only evaluated on a single hidden layer, fully connected network, on MNIST (Table 1). \n    \t1. MNIST is a very simple dataset, and a fully connected network is not well-suited for image classification. \n\t\t2. The differences between test performance of best and worst configurations are within 1%, and differences between He-Init and NN-GP-guided configs are within 0.25%. \n\t\t3. Notably, the likelihood-guided config performs worse than He-Init on the smallest training set size (10000). This is counter to the hypothesis that NN-GP likelihood evaluates the suitability of the NN prior, since we expect the quality of the prior to become more and more important as the training set becomes smaller. \n\t\t4. Finally, the likelihood-guided config of (3.6, 0) is at the corner of the hyper-parameter grid, which makes it unclear what would be observed if the grid of weight variances was extended further. \n\n    Due to all of the above, I find this experiment unconvincing. The hypothesis should be evaluated on a hyper-parameter grid that spans a large range of model performances (vs fractions of a percent), one where likelihood is not maximized at an edge of the grid, as well as considering other datasets (e.g. CIFAR-10), other hyper-parameters (e.g. depth), and more appropriate architectures (e.g. CNNs for image classification tasks). Note that there are a many other nuances that would need to be addressed to properly answer the the question **Q** (e.g. comparing computational expense of NN-GP-based search to those of grid/random/Bayesian search, including tuning the additional meta-hyper-parameters (NN-GP training set size, diagonal regularization), discussion of the additional technical complexity and feasibility of evaluating the NN-GP on practical architectures, how wide the finite network has to be in practice etc). It would be OK to not touch on these in the same paper, and only provide a convincing answer to a more specific question like **\"Q0: How well does NN-GP likelihood correlate with SGD-NN generalization?”**. Unfortunately, the paper doesn’t answer this either.\n        \n\t2. The idea itself is fairly straightforward, not particularly motivated theoretically (e.g. is there a bound on the difference in performance between NN-GP and SGD-trained NNs, perhaps in some toy cases?), and arguably not novel, as it is raised in one of the referenced papers ([Deep Neural Networks as Gaussian Processes, Lee et al 2018](https://arxiv.org/abs/1711.00165), page 6, second to last paragraph before section 3). This is to highlight that I see most of the potential value of this paper in answering the question **Q/Q0** (vs raising it), hence putting so much importance on my prior point. This would not have been an issue for me if the question was answered.\n\n### QUESTIONS:\n1. Could you please provide data (if it is present) for Table 1 for training sizes of 1000, 3000, 5000? These appear in Figure 3, but not Table 1.\n2. To tackle **Q0**, I encourage the authors to plot NN-GP likelihood vs test performance for all configurations that were considered (in addition to Table 1). To be clear, I’m afraid this would still not be sufficient to make a convincing case due to the limitations of the considered grid search discussed above.\n3. I am not certain/convinced with the point Figure 3 is making: is it that training curves of He-init and NN-GP-guided become more similar with increasing training set size? If so, I find the effect to be too small to be convincing on Figure 3 (perhaps zooming in  / measuring numerically could help), but I am also not sure why this would be an interesting/important metric to look at (vs test accuracy, as measured in Table 1).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Addressing the initialization issue in DNNs is an important topic, and the proposed approach is found by the reviewers to be interesting. However, the reviewers feel that to clearly promote this research beyond the 'proof of concept' phase, deeper investigation in multi-layer architectures would be required. This would raise the significance of the paper. Besides extending the study to deeper networks, the paper could also benefit from elaborate experiments to increase convincingness, in particular by addressing R4's concerns regarding robustness of performance e.g. on small dataset sizes. Finally, the methodology is sound and the authors clarify the significance of the ReLU associated covariance; however, overall the paper does not offer significant technical advancements that could make up for the shortcomings in the areas discussed above. 
Decision: Reject