Title: INFINITELY DEEP INFINITE-WIDTH NETWORKS

Abstract: Infinite-width neural networks have been extensively used to study the theoretical properties underlying the extraordinary empirical success of standard, finite-width neural networks. Nevertheless, until now, infinite-width networks have been limited to at most two hidden layers. To address this shortcoming, we study the initialisation requirements of these networks and show that the main challenge for constructing them is defining the appropriate sampling distributions for the weights. Based on these observations, we propose a principled approach to weight initialisation that correctly accounts for the functional nature of the hidden layer activations and facilitates the construction of arbitrarily many infinite-width layers, thus enabling the construction of arbitrarily deep infinite-width networks. The main idea of our approach is to iteratively reparametrise the hidden-layer activations into appropriately defined reproducing kernel Hilbert spaces and use the canonical way of constructing probability distributions over these spaces for specifying the required weight distributions in a principled way. Furthermore, we examine the practical implications of this construction for standard, finite-width networks. In particular, we derive a novel weight initialisation scheme for standard, finite-width networks that takes into account the structure of the data and information about the task at hand. We demonstrate the effectiveness of this weight initialisation approach on the MNIST, CIFAR-10 and Year Prediction MSD datasets.

1. INTRODUCTION:

While deep neural networks have achieved impressive empirical success on many tasks across a wide range of domains in recent years (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015; Sercu et al., 2015; Silver et al., 2016; Moravčík et al., 2017) , they remain hard to interpret black boxes whose performance crucially depends on many heuristics. In an attempt to better understand them, significant research effort has been directed towards examining the theoretical properties of these models with one important line of research focusing on the profound connections between infinite-width networks and kernel methods. In particular, Neal (1996) established a correspondence between single-layer infinite-width networks and Gaussian processes (GP) (Rasmussen, 2006) showing the equivalence of the prior over functions that is induced by the network and a GP with a particular covariance kernel. The appropriate covariance kernel has been analytically derived for a few particular activation functions and weight priors (Williams, 1997) .

Although there is a large body of research on infinite-width networks with surging recent interest in the topic (Hazan & Jaakkola, 2015; Lee et al., 2017; Matthews et al., 2018) , until now the construction of these networks has been limited to at most two hidden layers. In order to overcome this shortcoming and enable deep infinite-width networks, we propose a novel approach to the construction of networks with infinitely wide hidden layers. To the best of our knowledge, this is the first method that enables the construction of infinite-width networks with more than two hidden layers. The main challenge in constructing this type of networks lies in ensuring that the inner products between the hidden layer representations and the corresponding weights, which are both functions, are well-defined. In particular, this amounts to ensuring that the weights lie in the same function space as the hidden layer representations with which the inner product is taken. In other words, the weights connecting layers l and l + 1 need to be in the same function space as the activations of layer l. To construct the infinite-width layer l + 1, we need to construct infinitely many weights connecting layers l and l + 1 that fulfill this requirement, i.e. we need to define a probability distribution over the function space of activations of layer l. As the number of layers grows, the function spaces of activations grow increasingly more complex, thus making it increasingly more difficult to satisfy the requirements on the weights.

In order to tackle this challenge, we propose a principled approach to weight initialisation that automatically ensures that the weights are in the appropriate function space. The main idea of our approach is to make use of the canonical way of defining probability distributions over reproducing kernel Hilbert spaces (RKHS) and iteratively define the appropriate weight distributions facilitating the composition of arbitrarily many layers of infinite width. To this end, we first construct a kernel corresponding to each hidden layer and examine the associated RKHS of functions that is induced by this kernel. Next, for every layer, we establish a correspondence between the space of activations at a layer and the corresponding RKHS by reparametrising the hidden layer activations of a datapoint with the RKHS function corresponding to that point. Establishing this correspondence allows us to use a principled approach to defining probability distributions over RKHSs for constructing the appropriate sampling distribution of the weights in the infinite-width network.

We also examine some practical implications of this construction for the case of standard, finitewidth neural networks in terms of weight initialisation. Using Monte Carlo approximations, we derive a novel data-and task-dependent weight initialisation scheme for finite-width networks that incorporates the structure of the data and information about the task at hand into the network.

The main contributions of this paper are

• a novel approach to the construction of infinite-width networks that enables the construction of networks with arbitrarily many hidden layers of infinite-width,

• a hierarchy of increasingly complex kernels that capture the geometry and inductive biases of individual layers in the network,

• a novel weight initialisation scheme for deep neural networks with theoretical foundations established in the infinite-width case.

The rest of the paper is organised as follows. Section 2 discusses related work and Section 3 introduces our proposed approach to the construction of deep infinite-width networks. In Section 4, we showcase the practical implications of our theoretical contribution for the case of standard, finite-width deep networks. Section 5 discusses the experimental results followed by a conclusion in Section 6.

2. RELATED WORK:

Inspired by the groundbreaking work establishing the correspondence between single-layer neural networks and GPs (Neal, 1996; Williams, 1997) , there has been a recent resurgence of interest in examining the connections between neural networks and kernel methods. In particular, a large body of literature that is concerned with constructing kernels that mimic computations in neural networks has emerged outside of the "traditional" GP-neural network correspondence context. For example, Cho & Saul (2009) construct kernels that mimic the computations in neural networks with Sign and ReLU nonlinearities, while Daniely et al. (2016) discuss the duality in expressivity between compositional kernels and neural networks, and proposes a generalised construction approach for kernels corresponding to networks whose connectivity can be represented with a directed acyclic graph. Further, Gens & Domingos (2016) construct compositional kernels that correspond to convolutional networks, while additional connections between convolutional networks and kernels are discussed in Mairal et al. (2014) .

Using the multi-layer nature of neural networks within a kernel framework, Damianou & Lawrence (2013) and Duvenaud et al. (2014) discuss stacking GPs to construct deep Gaussian processes, while Wilson et al. (2016) and Al-Shedivat et al. (2017) use GPs with covariance kernels defined on the output of fully-connected and recurrent neural networks, respectively. Further examples include learning the kernel of an infinite-width neural network where the hidden layer outputs are binary (Heinemann et al., 2016) , learning a neural network by kernel alignment (Duan et al., 2018) and learning in a setting when the kernel can only be estimated through sampling (Livni et al., 2017) . Kernels have also been used to analyze neural networks (Montavon et al., 2011) and have been theoretically examined for consistency and universality (Steinwart et al., 2016) .

There has also been a lot of recent interest in examining the "traditional" GP-neural network correspondence. In particular, Hazan & Jaakkola (2015) construct kernels for infinite-width neural networks with up to two hidden layers and arbitrary nonlinearities. In particular, a GP with a general covariance kernel is used for sampling the weights between the two hidden layers, and the hidden layer activations are assumed to be L 2 -functions. For arbitrary network non-linearities and data distributions, the hidden layer activations do not exhibit this level of regularity. Furthermore, since the GP covariance kernel is not chosen in accordance to the structure hidden layer activations, the weights will not have the appropriate sampling distribution, and the resulting network will, in general, be ill-defined. Two recent papers (Lee et al., 2017; Matthews et al., 2018) discussed the connections between Gaussian processes and deep neural networks for increasing hidden layer width. In particular, Lee et al. (2017) derive a recurrence relation for the kernel of a deep network and use it within a GP to perform fully Bayesian inference for regression, while Matthews et al. ( 2018) establish theoretical results characterising the convergence in distribution of deep networks to the corresponding GP. Our work differs from these two contributions in three key aspects. First, the main motivation for our approach is enabling the construction of deep infinite-width neural networks as opposed to examining the behaviour of deep neural networks as the width of its hidden layers increases. Second, we study and discuss the initialisation requirements of infinite-width neural networks, while Lee et al. (2017) and Matthews et al. (2018) use standard initialisations in finite-width neural networks and do not properly address the changing nature of the initialisation requirements of the weights as the width of the hidden layers goes to infinity. In particular, in the limit of infinite width, the weights cannot be sampled from a normal distribution as this causes the inner product with the infinite-width hidden layer to be ill-defined. Third, as we follow different goals and approaches, the definition and utilisation of kernels differ. In particular, we define a kernel on the space of the activations of a layer, whereas Lee et al. (2017) and Matthews et al. ( 2018) define a kernel on the space of pre-activations of that layer. This important distinction allows us to construct the weight distributions needed for constructing deep infinite-width networks and results in the kernels encoding different aspects of the neural network behaviour.

3. DEEP INFINITE-WIDTH NETWORKS:

In this section, we present a novel method for the construction of infinite-width networks. To the best of our knowledge, this is the first approach that enables the construction of arbitrarily deep infinite-width networks. Previous work addressing infinite-width networks was either limited to at most two layers of infinite width (Hazan & Jaakkola, 2015) or examined limits of deep finite-width networks and did not appropriately account for the changing nature of the weights in the limit of infinite width (Lee et al., 2017; Matthews et al., 2018) . In particular these approaches ignore the problem that the weights connecting two adjacent infinite-width layers cannot be sampled the same way as in finite-width networks, i.e. from N (0, σ 2 w ), as this does not yield a well-defined inner product with the incoming layer activations.

3.1. DIFFICULTY OF CONSTRUCTING NETWORKS WITH MORE THAN TWO LAYERS:

In this subsection, we examine the initialisation requirements of infinite-width networks. In particular, we show that the main challenge in constructing these networks is specifying an appropriate sampling distributions for the weights that will ensure that the network as a whole is well-defined. Given an input x ∈ X ⊆ R d , we compute the activation of neuron i in the first layer as

EQUATION

with the layer non-linearity f 1 applied elementwise and w 0 i the weights connecting the input layer to the i-th neuron in the first layer. Here, we make the dependence of the activations φ 1,x on the connecting weights explicit as it allows us to more effectively reason about the initialisation requirements. For the first layer representation φ 1,x to be well-defined, φ 1,x (w 0 i ) needs to be welldefined for every neuron i, i.e. the inner product w 0 i , x R d needs to be well-defined for every i. For the inner products to be well-defined, the weights w 0 i need to lie in the same space as x, i.e. in R d . Thus, in order to construct a first infinite-width layer that is well-defined, we need to sample infinitely many weights w 0

• from R d . For this purpose, we can choose any probability measure ν 0 defined on R d . Let us denote by Φ 1 the space of the first layer activations, i.e.

Φ 1 = Span {φ 1,x | x ∈ X }), with f, g Φ1 = f (w)g(w)dν 0 (w) for f, g ∈ Φ 1

the inner product and the bar denoting the completion of the set with respect to the norm induced by the inner product. We note that Φ 1 is a space of functions as the first layer has infinite width. For the activation of neuron i in the second layer, we can write

φ 2,x (w 1 i ) = f 2 w 1 i , φ 1,x Φ1

with the layer non-linearity f 2 applied elementwise and w1 i the weights connecting the first layer to the i-th neuron in the second layer. In order for the second layer activations φ 2,x to be well-defined, φ 2,x (w 1 i ) needs to be well-defined for every i, i.e. the inner products w 1 i , φ 1,x Φ1 need to be welldefined for every i. In order for this to be the case, the weights w 1 i have to lie in the same space as φ 1,x for every i. Since φ 1,x ∈ Φ 1 , we need to construct a sampling distribution over Φ 1 for the weights connecting the first and second layers. As Φ 1 is a space of functions defined over R d , a natural choice for the sampling distribution of w 1

• is a GP. For constructing infinite-width networks with at most two layers, Hazan & Jaakkola (2015) endowed this GP with a general covariance function. However, as they did not take into account the structure of the underlying space Φ 1 , this GP will, in general, not define a sampling distribution over Φ 1 , thus yielding a network which is ill-defined as the weights will not sampled from the appropriate space.

Analogously to the previous two layers, in order to construct a third infinite-width layer, we need to define the weight distribution over the space of functions of functions over R d that is Φ 2 = Span({φ 2,x | x ∈ X }) as to ensure that taking inner products with φ 2,x is well-defined. Unfortunately, until now it was not clear how to construct this sampling distribution.

In summary, the main challenge in constructing infinite-width networks is defining appropriate sampling distributions for the weights over increasingly more complex function spaces that are induced by the hidden layer activations to ensure that the network is well-defined. In particular, in order to construct layer l + 1 of infinite-width, we need to construct a distribution over the function space of activations at layer l, thus making the inner products between the weights and the l-th layer activations well-defined. Constructing the appropriate weight distributions is a highly non-trivial undertaking for networks with more than two layers of infinite-width as it is not clear how to go about constructing distributions over spaces of functions of functions.

3.2. CONSTRUCTING DEEP INFINITE-WIDTH NETWORKS:

To address the above difficulty in constructing the weight distributions in infinite-width networks, we propose a principled approach to weight initialisation that automatically ensures that the weights are drawn from the appropriate function spaces. This initialisation scheme allows us to overcome the current limitation of just two hidden layers and construct deep infinite-width networks with arbitrarily many hidden layers. The main idea of our approach is to use the canonical way of defining probability distributions over RKHSs for constructing appropriate sampling distributions of the weights. To this end, we construct at every hidden layer an associated kernel and reparametrise the hidden layer activations into the RKHSs that are induced by these kernels. As a consequence, this yields a hierarchy of increasingly more complex kernels capturing the geometry and inductive biases of the network.

As before, let x ∈ X ⊆ R d be an input and φ 1,x (w i 0 ) = f 1 ( w i 0 , x R d ) the activation of neuron i in the first layer. Associated with this layer, we define a kernel by taking the inner product between the activations. Specifically, we define

k 1 x, x = φ 1,x , φ 1,x Φ1 = φ 1,x w φ 1,x w dν 0 w (2)

and suppress the dependence of k 1 on the non-linearity f 1 when it does not lead to notational ambiguity. 1 This kernel captures the geometry of the space of first activations and compactly encodes the inductive biases of that layer.

By the Moore-Aronszajn theorem (Berlinet & Thomas-Agnan, 2011) , the kernel k 1 induces a corresponding RKHS H k1 = Span {k 1 (•, x)| x ∈ X } which is the closure of the span of so-called canonical feature maps k 1 •, x of k 1 . Associated with H k1 , there is an inner product •, • H k 1 and the reproducing property holds, i.e.

∀x ∈ X , ∀f ∈ H k1 f, k 1 (•, x) H k 1 = f (x) ∀x, x ∈ X k 1 x, x = k 1 •, x , k 1 •, x H k 1 .

Now, in order to construct a second layer of infinite-width, we need to define a distribution over Φ 1 for the weights connecting the first and the second layer. To this end, we use the structure of the induced RKHS H k1 and the fact that defining a probability distribution over an RKHS can be done in a principled way using the corresponding kernel. The following proposition and lemma summarise our weight initialisation scheme for infinite-width networks. Proposition 1. Let x, x ∈ R d be inputs to a network with l infinite-width layers, k l be the kernel corresponding to the l-th layer and H k l the induced RKHS. We can extend the network with an (l + 1)-th layer of infinite width by sampling the weights w l connecting the l-th and (l + 1)-th layers from a Gaussian process with zero mean function and covariance function

EQUATION

Lemma 1 (Network Reparametrisation). Given the l-th layer activations Φ l = Span {φ l,x |x ∈ X }), the associated kernel k l and the induced RKHS H k l , the mapping

U l : Φ l → H k l with φ l,x → k l •, x

is an isometric isomorphism between Φ l and H k l .

Proofs of the proposition and lemma are given in the Supplementary Material, but we briefly outline here the main intuitions behind them. In particular, our proposed construction of an infinite-width network proceeds iteratively layer by layer. Having constructed the l-th layer, we construct the kernel k l associated with that layer. This kernel induces an RKHS H k l . We then perform a reparametrisation of the l-th layer activations Φ l into H k l according to Lemma 1. Having identified Φ l with H k l , we construct the appropriate distribution of the weights connecting layers l and l + 1 as the canonical distribution over H k l . Finally, to construct layer l + 1, we sample infinitely many of the connecting weights according to (3).

4. INITIALISATION BY PROJECTION:

Having established a principled construction approach for infinite-width networks, we now turn our attention to examining the consequences of this construction for standard, finite-width networks.

In particular, by reasoning about finite-width networks as Monte Carlo approximations of their infinite-width counterparts, we derive a novel weight initialisation method based on the above weight initialisation scheme for infinite-width networks and examine its structural properties.

In the finite-width case, we can simplify our notation. In particular, given an input to the network x ∈ X ⊆ R d , we denote by h l (x) the output of layer l and compute its i-th entry recursively from its infinite-width counterpart φ l,x (w l i ) as

[h l (x)] i := φl,x (w l-1 i ) = f l ( w l-1 i , φl-1,x R N l-1 ) = f l ( w l-1 i , h l-1 (x) R N l-1 )

where• refers to the finite-width approximation of the infinite-width quantity. In particular, in the finitewidth case, we have Φl ⊆ R N l with N l the width of the l-th layer and Φl = Span({ φl,x |x ∈ X }). Concatenating all the N l weight vectors that connect layers l -1 and l, and lie in R N l-1 , we get the weight matrix W l-1 ∈ R N l ×N l-1 .

Most common stochastic initialisation approaches are data-and task-agnostic and draw the weights from a Gaussian distribution with zero mean and appropriately scaled variance. In contrast to that, we obtain a data-and task-dependent approach to initialisation we term Weight Initialisation with Infinite Networks (Win-Win) by taking Monte Carlo approximations of the weight initialisation for infinite-width networks. In particular, we approximate the integrals used for computing the kernels and GP covariance from equations ( 2) and (3) through sampling, i.e.

kl (x, x ) = 1 N l N l n=1 f ( x, w l n R N l-1 )f ( x , w l n R N l-1 ) with kl (•, x) i = 1 √ N l f ( x, w l i R N l-1 ) C l (x, x ) ≈ 1 M l M l j=1 k l (x, ξ j )k l (x , ξ j ) ≈ 1 M l M l j=1 kl (x, ξ j ) kl (x , ξ j ) =: Ĉl (x, x ).

While the number of samples used in the approximation corresponds to the dimensionality of the feature expansion, i.e. the size of the corresponding finite-width layers, we can also think of these representations as random feature expansions of a particular kernel thus establishing a connection to Rahimi & Recht (2008) . In the finite-width case, we have w l i ∼ GP(0, Ĉl ) which can be written as

w l i = M m=1 α m,i h l (ξ m,i ) with α m,i ∼ N 0, 1 M

where the coefficients α m,i are mixing weights and ξ m,i are selected training points.

This approach encourages us to think about the interaction between the weights and the hidden layer representations in finite-width networks from a new perspective. In particular, the rows of the weight matrix at layer l are initialised as weighted linear combinations of the activations of some points in l-th layer. This facilitates an intuitive interpretation of a neuron i in layer l + 1 measuring the alignment of the data with the subspace spanned by {ξ m,i } m . We propose three different approaches to choosing these subspaces. First, we can choose the points randomly from the training data at every layer; this corresponds to projecting the data onto random subspaces of the training distribution. We denote this approach as Win-Win random. Next, guided by the principle of disentangling the factors of variation in the data, we propose two structured approaches to the subspace selection that incorporate the structure of the data and information about the task at hand directly into the weights already at initialisation. In particular, we disentangle the data manifold by making similar objects increasingly more similar and dissimilar objects increasingly more dissimilar as we move from the lower to the higher layers. For the data and task at hand, we propose two ways to uncover the "disentangled" directions: using k-means clustering and class information, respectively. Pre-training neural networks has a long history of investigation (Hinton et al., 2006; Erhan et al., 2010) . Interestingly, the practical methods we propose could be seen as particular kinds of pre-training principally derived from the initialisation of an infinitely-wide neural network. The methods we propose initialise the weights so that the learning dynamics are subtly modified with a bias towards the data manifold.

K-means-based subspace selection. For weights connecting layers l and l + 1, we select the subspaces by clustering the activations of the training data at layer l into N l+1 clusters, i.e. one cluster is assigned to every neuron in the subsequent layer. We pick p points from cluster i, i.e. {ξ pos m,i } p m=1 , and n points from each of the remaining clusters, i.e. {ξ j,neg m,i } n m=1 for j ≤ N l+1 , j = i, and compute the weight vector w l i connecting layer l with neuron i in layer l + 1 as

EQUATION

)

Class-based subspace selection. If the task at hand is classification, we can choose the subspaces in a manner that is more aligned with this end goal. Instead of clustering the activations, we use the class labels as cluster assignments. Specifically, we associate every neuron i in layer l + 1 with a class c and pick p points {ξ pos m,i } p m=1 from that class. We also pick {ξ j,neg m,i } n m=1 points from each of the remaining classes j = c and compute the weight vector w l i as

w l i = p m=1 α m,i h l (ξ pos m,j ) - j =c n m=1 α m,j,i h l (ξ j,neg m,i ).

In both subspace selection methods, points that are from cluster i should activate neuron i more than points from other clusters since we are measuring the alignment to points from that cluster, while substracting the alignment scores of all other clusters. Thus, we are making the points from cluster i more similar to each other and less similar to points from all other clusters.

5. EXPERIMENTAL RESULTS:

We compared our new initialisation scheme Win-Win to 4 commonly used initialisation schemes, namely Xavier (Glorot & Bengio, 2010) , LeCun (LeCun et al., 2012) , Kaiming (He et al., 2015) and SntDefault (Ioffe & Szegedy, 2015) , the default initialiser in Sonnet, using the MNIST dataset (LeCun et al., 1998) , the CIFAR-10 dataset (Krizhevsky & Hinton, 2009) and on the Year Prediction MSD dataset (Bertin-Mahieux et al., 2011) . For all experiments, we use the validation set for hyperparameter selection and early stopping, and the Adam optimizer (Kingma & Ba, 2014) with the default hyperparameters and a fixed learning rate for the whole of training. Throughout, we used the ReLU non-linearity. Error bars are one standard deviation of the mean and are computed using an unbiased estimate of the variance for 12 different random seeds. We note that we did not use any advanced techniques, such as data augmentation or learning rate scheduling, in order to disentangle the effect of initialisation on network performance from the effects of other training heuristics.

MNIST. As a sanity check, we test our method on the benchmark dataset MNIST which encodes handwritten digits from 0 to 9 as grayscale images of size 28 × 28 pixels with a train/valid/test split of 50, 000/10, 000/10, 000 points. We train a 2-layer fully-connected architecture with 800 hidden units. We choose this architecture as they have achieved good performance as reported on http://yann.lecun.com/exdb/mnist/. We test the three different methods for subspace selection in Win-Win -random, class and kmeans. As can be seen from Table 1 , random and kmeans-based Win-Win are competitive to all the competing methods. Class-based Win-Win performs worse than other methods which we posit is due to the class information only being informative for initialising the weights connecting to the output layer and not for the weights incoming to the first hidden layer. 2.17 ± 0.21%

In the Supplementary Material, we also study the evolution of activation patterns for the different initialisation methods. In particular, we look at the activations of 100 randomly sampled points from each class immediately after initialisation and during training. For the standard initialisation approaches, the activation pattern across different neurons and classes is rather sparse, see Figures 1 2 3 4 . At initialisation only very few neurons from the hidden layer contribute to the calculation of the logits. As training progresses, more neurons from the hidden layer get activated, but even after the network has been completely trained, we see that a large number of neurons does not fire for any class. On the other hand, with class-based Win-Win initialisation, we see that neurons assigned to class c fire more prominently for that class than for other classes, while the logits correspond to a uniform prior over the neurons for different classes. As training progresses, we see that this diagonal structure in the hidden layer is roughly preserved without specific constraints.

CIFAR-10. This dataset consists of 60, 000 images of size 32 × 32 × 3 pixels with a train/valid/test split of 40 000/10 000/10 000. As is common practice, we pre-process the images by scaling them to the range [-1, 1]. For the model, we use a deep convolutional neural network with a fully-connected linear layer as the output layer; details of the architecture are provided in the Supplementary Material. We trained the network for 400 epochs. We only initialised the fully-connected layer according to the different initialisation schemes and the rest initialised with the default initialiser (Ioffe & Szegedy, 2015) . We do this in order to demonstrate that Win-Win is beneficial also in situations when it can only be applied to features derived from some transformation of the input data (in this case structured random projections) and not directly to the input data. As can be seen Year Prediction MSD dataset. In addition to the two classification problems summarised above, we assessed the performance of our new initialisation scheme also on a regression task using the Year Prediction MSD dataset (Bertin-Mahieux et al., 2011) . In this task, the goal is to estimate the year in which a song was released based on a 90-dimensional input vector of audio features. We split the available data into training, validation and test sets containing 400 000, 63 715 and 51 630 data points, respectively. In a pre-processing step, we then computed the mean and standard deviation of features and labels of the training set, and used these quantities to shift and rescale all three datasets so that the distribution of data points is approximately standard normal.

For comparison, we trained two different, fully-connected architectures with one or two hidden layers followed by a single-unit output layer for a total of 150 epochs with the objective to minimise the mean squared error (MSE). As a natural choice for class-based Win-Win initialisation, we set the number of units in each hidden layer to 89 which corresponds to the number of distinct labels in the training set. As a measure of performance, we use the root mean squared error (RMSE) achieved on the test set (Lakshminarayanan et al., 2017) . The experimental results are summarised in Table 3 . We can clearly see that Win-Win initialisation outperforms all other schemes in case of a single hidden layer and is comparable to the best performing scheme (SntDefault) for two hidden layers. 

6. CONCLUSION:

In this paper, we have studied the initialisation requirements of infinite-width networks and have shown that the main challenge in constructing these networks lies in defining the appropriate sampling distributions of the weights. To address this problem, we have presented a novel method for the construction of infinite-width networks that, unlike previous approaches, enables the construction of deep infinite-width networks with arbitrarily many hidden layers. In particular, we have proposed a principled approach to weight initialisation using the theory of reproducing kernel Hilbert spaces. In order to appropriately account for the functional form of the hidden layer activations and to facilitate the construction of arbitrarily many infinite-width layers, we proposed to construct the sampling distributions of the weights at every hidden layer as Gaussian processes with specific covariance kernels that take into account the geometry of the underlying space of activations. To achieve this, we have constructed a hierarchy of kernels that capture the geometry and inductive biases of individual layers in the neural network. Furthermore, using Monte Carlo approximations, we have examined the practical implications of this construction for standard, finite-width networks. In particular, we have derived a novel data-and task-dependent weight initialisation method for this type of network and showcased its competitive performance on three diverse datasets.

7. SUPPLEMENTARY MATERIAL:

7.1 PROOFS For completeness, we restate the proposition and lemma and provide the corresponding proofs.

7.1.1. PROPOSITION 1:

Proposition 1. Let x, x ∈ R d be inputs to an network with l infinite-width layers, k l be the kernel corresponding to the l-th layer with k l (•, x) the corresponding canonical feature maps and H k l the induced RKHS. We can extend the network with an (l + 1)-th layer of infinite width by sampling the weights w l connecting the l-th and (l + 1)-th layers from a Gaussian process with zero mean function and covariance function

C l (w, w ) = k l (w, w l-1 )k l (w , w l-1 ) dν l-1 (w l-1 ) with w, w ∈ Φ l-1

Then w l ∈ Φ l and φ l,x (w l ) = f ( w l , φ l-1,x Φ l ) is well defined.

Proof of Proposition.:

We prove the claim of this proposition by mathematical induction. We first need to establish a correspondence between Φ 1 and H k1 and construct an isometric isomorphism (Coxeter, 1963) between Φ 1 and H k1 . To this end, we invoke Lemma 1 for the first hidden layer. The isometric isomorphism U 1 identifies Φ 1 with H k1 and allows us to replace Φ 1 with the easier to work with H k1 as the geometry remains unchanged. The transformations involved in the proof are depicted as below.

Φ 1 H k1 φ 1,x k 1 (•, x) U 1 U 1

Using this reparametrization, instead of constructing a distribution for the weights over Φ 1 in order to construct a second layer of infinite width, the problem simplifies to constructing a distribution for the weights over H k1 . In particular, we use the canonical probability distribution over H k1 to define the sampling distribution for the weights between the first and second infinite-width layer.

Specifically, a probability distribution over H k1 can be canonically constructed as a Gaussian process with zero mean function and covariance function C 1 (w, w ) = k 1 (w, w 0 )k 1 (w, w 0 ) dν 0 (w 0 ) with w, w ∈ Φ 1 .

This special covariance structure ensures that a GP with this covariance function is a probability distribution over H k1 (Aronszajn, 1950) .

To construct a second hidden layer of infinite-width, we need to sample infinitely many weights w 1 connecting the first and second layers from ν 1 = GP (0, C 1 ). This construction ensures that the inner product between the first layer representation φ 1,x and the weights connecting the first and second layer is well defined. This, in turn, ensures that the second layer representation φ 2,x given by φ 2,k1(•,x) (w 1 ) = f w 1 , k 1 (•, x) H k 1 is well defined. We note that φ 2,x is a function over samples from the GP ν 1 , i.e. a function of functions over R d . Associated with the second layer, we define the corresponding kernel as k 2 (φ 1,x , φ 1,x ) = φ 2,x , φ 2,x Φ2

(5)

with Φ 2 = Span({φ 2,x |x ∈ X }) and the induced RKHS H k2 = Span({k 2 (•, φ 1,x )|x ∈ X }. The correspondence between these two spaces is given by the isometric isomorphism U 2 : Φ 2 → H k2 with U 2 (φ 2,x ) = k 2 (•, φ 1,x ) using Lemma 1.

neurons. For the first hidden layer where there are more than 10 neurons, we construct consecutive buckets of neurons of one tenth of the layer width and average over these. 

:

Without loss of generality, we have assumed that Φ1 ∈ L2(R d ). If Φ1 consists of less smooth space of functions than L2(R d ), we just define the kernel k1 such that the functions k1(•, x) for x ∈ X are L2-functions, e.g. as k1(x, x ) = exp(-( φ1,x, φ 1,x Φ 1 )/σ

).

Figures and Tables:

Figure 1: Softmaxed activations of the hidden and output layer for LeCun initialisation at epochs 0, 20 and 400 for a network with 800 hidden units.

Figure 2: Softmaxed activations of the hidden and output layer for Xavier initialisation at epochs 0, 20 and 400 for a network with 800 hidden units.

Figure 3: Softmaxed activations of the hidden and output layer for Kaiming initialisation at epochs 0, 20 and 400 for a network with 800 hidden units.

Figure 4: Softmaxed activations of the hidden and output layer for SntDefault initialisation at epochs 0, 20 and 400 for a network with 800 hidden units.

Figure 5: Softmaxed activations of the hidden and output layer for Win-Win class initialisation at epochs 0, 20 and 400 for a network with 800 hidden units.

Final classification error on the MNIST test set.

Table content: <table><tr><td>Method</td><td>Error</td></tr><tr><td>SntDefault</td><td>1.63 ± 0.20 %</td></tr><tr><td>LeCun</td><td>1.68 ± 0.24 %</td></tr><tr><td>Xavier</td><td>1.85 ± 0.17 %</td></tr><tr><td>Kaiming</td><td>1.63 ± 0.23 %</td></tr><tr><td>Win-Win [random]</td><td>1.67 ± 0.22%</td></tr><tr><td colspan="2">Win-Win [k-means] 1.58 ± 0.24 %</td></tr><tr><td>Win-Win [class]</td><td/></tr></table>

Table 2, the Win-Win initialisation clearly outperforms all other methods by a significant margin.

Table content: <table/>

Classification accuracy on the CIFAR10 test set.

Table content: <table><tr><td>Method</td><td>SntDefault</td><td>Xavier</td><td colspan="2">LeCun Kaiming Win-Win</td></tr><tr><td>Test Accuracy</td><td colspan="3">85.83% 86.38% 86.28%</td><td>86.08% 87.05 %</td></tr></table>

Regression RMSE values for the Year Prediction MSD dataset. We used SntDefault initialisation for the output layer in case of Win-Win initialisation.

Table content: <table><tr><td>Method</td><td>1 hidden layer</td><td>2 hidden layers</td></tr><tr><td>SntDefault</td><td>8.9301 ± 0.0033</td><td>8.8798 ± 0.0072</td></tr><tr><td>LeCun</td><td>8.9484 ± 0.0024</td><td>8.9414 ± 0.0086</td></tr><tr><td>Xavier</td><td>8.9376 ± 0.0044</td><td>8.8948 ± 0.0064</td></tr><tr><td>Kaiming</td><td>8.9448 ± 0.0039</td><td>8.9290 ± 0.0044</td></tr><tr><td colspan="3">Win-Win [random/SntDefault] 8.8902 ± 0.0022 8.8693 ± 0.0060</td></tr><tr><td>Win-Win [class/SntDefault]</td><td>8.8904 ± 0.0023</td><td>8.8867 ± 0.0057</td></tr></table>

Peer review (iclr2019):
 Pros: This paper uses kernel mappings between any two layers for weight initialisation. Using the representer theorem, a proper distribution for weights is constructed in H_{k_i} instead of being learned by .phi_i, and then is formulated as a GP. Cons: However, there are some key issues. 1. The so-called “infinite width” is just yielded by kernels in RKHS for weight initialization. For practical implementation, the authors use this scheme with random Fourier features to construct finite-width network. A key issue is that how to guarantee that the approximated weights are still in the same space? For example, weights can be in RKHS, but their approximation might be not in RKHS. See in [S1] for details. [S1] Generalization Properties of Learning with Random Features, NIPS 2017. 2. Experimental part is not very convincing. First, the authors just compare different initialization schemes. The used architectures are simple and not representative. Second, the overall performance is not satisfactory, and the compared classification datasets are quite small. Overall, the experimental results are inadequate and unconvincing. Summary: The paper attempts to proposal a weight initialization scheme to enable infinite deep infinite-width networks. However, there are some key issues not address such as whether the approximated weights are still in the same space and the limited experimental results. Response to rebuttal: The authors have addressed my question about the weights being still in the same RKHS. I still think the motivation and experiments are not very satisfactory. Therefore the paper is very borderline. However, I would like to bump my rating a bit higher.