Infinitely Deep Infinite-Width Networks

1 INTRODUCTION . While deep neural networks have achieved impressive empirical success on many tasks across a wide range of domains in recent years ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ; Szegedy et al. , 2015 ; Sercu et al. , 2015 ; Silver et al. , 2016 ; Moravčík et al. , 2017 ) , they remain hard to interpret black boxes whose performance crucially depends on many heuristics . In an attempt to better understand them , significant research effort has been directed towards examining the theoretical properties of these models with one important line of research focusing on the profound connections between infinite-width networks and kernel methods . In particular , Neal ( 1996 ) established a correspondence between single-layer infinite-width networks and Gaussian processes ( GP ) ( Rasmussen , 2006 ) showing the equivalence of the prior over functions that is induced by the network and a GP with a particular covariance kernel . The appropriate covariance kernel has been analytically derived for a few particular activation functions and weight priors ( Williams , 1997 ) . Although there is a large body of research on infinite-width networks with surging recent interest in the topic ( Hazan & Jaakkola , 2015 ; Lee et al. , 2017 ; Matthews et al. , 2018 ) , until now the construction of these networks has been limited to at most two hidden layers . In order to overcome this shortcoming and enable deep infinite-width networks , we propose a novel approach to the construction of networks with infinitely wide hidden layers . To the best of our knowledge , this is the first method that enables the construction of infinite-width networks with more than two hidden layers . The main challenge in constructing this type of networks lies in ensuring that the inner products between the hidden layer representations and the corresponding weights , which are both functions , are well-defined . In particular , this amounts to ensuring that the weights lie in the same function space as the hidden layer representations with which the inner product is taken . In other words , the weights connecting layers l and l + 1 need to be in the same function space as the activations of layer l. To construct the infinite-width layer l + 1 , we need to construct infinitely many weights connecting layers l and l + 1 that fulfill this requirement , i.e . we need to define a probability distribution over the function space of activations of layer l. As the number of layers grows , the function spaces of activations grow increasingly more complex , thus making it increasingly more difficult to satisfy the requirements on the weights . In order to tackle this challenge , we propose a principled approach to weight initialisation that automatically ensures that the weights are in the appropriate function space . The main idea of our approach is to make use of the canonical way of defining probability distributions over reproducing kernel Hilbert spaces ( RKHS ) and iteratively define the appropriate weight distributions facilitating the composition of arbitrarily many layers of infinite width . To this end , we first construct a kernel corresponding to each hidden layer and examine the associated RKHS of functions that is induced by this kernel . Next , for every layer , we establish a correspondence between the space of activations at a layer and the corresponding RKHS by reparametrising the hidden layer activations of a datapoint with the RKHS function corresponding to that point . Establishing this correspondence allows us to use a principled approach to defining probability distributions over RKHSs for constructing the appropriate sampling distribution of the weights in the infinite-width network . We also examine some practical implications of this construction for the case of standard , finitewidth neural networks in terms of weight initialisation . Using Monte Carlo approximations , we derive a novel data- and task-dependent weight initialisation scheme for finite-width networks that incorporates the structure of the data and information about the task at hand into the network . The main contributions of this paper are • a novel approach to the construction of infinite-width networks that enables the construction of networks with arbitrarily many hidden layers of infinite-width , • a hierarchy of increasingly complex kernels that capture the geometry and inductive biases of individual layers in the network , • a novel weight initialisation scheme for deep neural networks with theoretical foundations established in the infinite-width case . The rest of the paper is organised as follows . Section 2 discusses related work and Section 3 introduces our proposed approach to the construction of deep infinite-width networks . In Section 4 , we showcase the practical implications of our theoretical contribution for the case of standard , finite-width deep networks . Section 5 discusses the experimental results followed by a conclusion in Section 6 . 2 RELATED WORK . Inspired by the groundbreaking work establishing the correspondence between single-layer neural networks and GPs ( Neal , 1996 ; Williams , 1997 ) , there has been a recent resurgence of interest in examining the connections between neural networks and kernel methods . In particular , a large body of literature that is concerned with constructing kernels that mimic computations in neural networks has emerged outside of the `` traditional '' GP-neural network correspondence context . For example , Cho & Saul ( 2009 ) construct kernels that mimic the computations in neural networks with Sign and ReLU nonlinearities , while Daniely et al . ( 2016 ) discuss the duality in expressivity between compositional kernels and neural networks , and proposes a generalised construction approach for kernels corresponding to networks whose connectivity can be represented with a directed acyclic graph . Further , Gens & Domingos ( 2016 ) construct compositional kernels that correspond to convolutional networks , while additional connections between convolutional networks and kernels are discussed in Mairal et al . ( 2014 ) . Using the multi-layer nature of neural networks within a kernel framework , Damianou & Lawrence ( 2013 ) and Duvenaud et al . ( 2014 ) discuss stacking GPs to construct deep Gaussian processes , while Wilson et al . ( 2016 ) and Al-Shedivat et al . ( 2017 ) use GPs with covariance kernels defined on the output of fully-connected and recurrent neural networks , respectively . Further examples include learning the kernel of an infinite-width neural network where the hidden layer outputs are binary ( Heinemann et al. , 2016 ) , learning a neural network by kernel alignment ( Duan et al. , 2018 ) and learning in a setting when the kernel can only be estimated through sampling ( Livni et al. , 2017 ) . Kernels have also been used to analyze neural networks ( Montavon et al. , 2011 ) and have been theoretically examined for consistency and universality ( Steinwart et al. , 2016 ) . There has also been a lot of recent interest in examining the `` traditional '' GP-neural network correspondence . In particular , Hazan & Jaakkola ( 2015 ) construct kernels for infinite-width neural networks with up to two hidden layers and arbitrary nonlinearities . In particular , a GP with a general covariance kernel is used for sampling the weights between the two hidden layers , and the hidden layer activations are assumed to be L2-functions . For arbitrary network non-linearities and data distributions , the hidden layer activations do not exhibit this level of regularity . Furthermore , since the GP covariance kernel is not chosen in accordance to the structure hidden layer activations , the weights will not have the appropriate sampling distribution , and the resulting network will , in general , be ill-defined . Two recent papers ( Lee et al. , 2017 ; Matthews et al. , 2018 ) discussed the connections between Gaussian processes and deep neural networks for increasing hidden layer width . In particular , Lee et al . ( 2017 ) derive a recurrence relation for the kernel of a deep network and use it within a GP to perform fully Bayesian inference for regression , while Matthews et al . ( 2018 ) establish theoretical results characterising the convergence in distribution of deep networks to the corresponding GP . Our work differs from these two contributions in three key aspects . First , the main motivation for our approach is enabling the construction of deep infinite-width neural networks as opposed to examining the behaviour of deep neural networks as the width of its hidden layers increases . Second , we study and discuss the initialisation requirements of infinite-width neural networks , while Lee et al . ( 2017 ) and Matthews et al . ( 2018 ) use standard initialisations in finite-width neural networks and do not properly address the changing nature of the initialisation requirements of the weights as the width of the hidden layers goes to infinity . In particular , in the limit of infinite width , the weights can not be sampled from a normal distribution as this causes the inner product with the infinite-width hidden layer to be ill-defined . Third , as we follow different goals and approaches , the definition and utilisation of kernels differ . In particular , we define a kernel on the space of the activations of a layer , whereas Lee et al . ( 2017 ) and Matthews et al . ( 2018 ) define a kernel on the space of pre-activations of that layer . This important distinction allows us to construct the weight distributions needed for constructing deep infinite-width networks and results in the kernels encoding different aspects of the neural network behaviour . 3 DEEP INFINITE-WIDTH NETWORKS . In this section , we present a novel method for the construction of infinite-width networks . To the best of our knowledge , this is the first approach that enables the construction of arbitrarily deep infinite-width networks . Previous work addressing infinite-width networks was either limited to at most two layers of infinite width ( Hazan & Jaakkola , 2015 ) or examined limits of deep finite-width networks and did not appropriately account for the changing nature of the weights in the limit of infinite width ( Lee et al. , 2017 ; Matthews et al. , 2018 ) . In particular these approaches ignore the problem that the weights connecting two adjacent infinite-width layers can not be sampled the same way as in finite-width networks , i.e . from N ( 0 , σ2w ) , as this does not yield a well-defined inner product with the incoming layer activations . 3.1 DIFFICULTY OF CONSTRUCTING NETWORKS WITH MORE THAN TWO LAYERS . In this subsection , we examine the initialisation requirements of infinite-width networks . In particular , we show that the main challenge in constructing these networks is specifying an appropriate sampling distributions for the weights that will ensure that the network as a whole is well-defined . Given an input x ∈ X ⊆ Rd , we compute the activation of neuron i in the first layer as φ1 , x ( w 0 i ) = f1 ( 〈w0i , x〉Rd ) ( 1 ) with the layer non-linearity f1 applied elementwise and w0i the weights connecting the input layer to the i-th neuron in the first layer . Here , we make the dependence of the activations φ1 , x on the connecting weights explicit as it allows us to more effectively reason about the initialisation requirements . For the first layer representation φ1 , x to be well-defined , φ1 , x ( w0i ) needs to be welldefined for every neuron i , i.e . the inner product 〈w0i , x〉Rd needs to be well-defined for every i . For the inner products to be well-defined , the weights w0i need to lie in the same space as x , i.e . in Rd . Thus , in order to construct a first infinite-width layer that is well-defined , we need to sample infinitely many weights w0· from Rd . For this purpose , we can choose any probability measure ν0 defined on Rd . Let us denote by Φ1 the space of the first layer activations , i.e . Φ1 = Span ( { φ1 , x| x ∈ X } ) , with 〈f , g〉Φ1 = ∫ f ( w ) g ( w ) dν0 ( w ) for f , g ∈ Φ1 the inner product and the bar denoting the completion of the set with respect to the norm induced by the inner product . We note that Φ1 is a space of functions as the first layer has infinite width . For the activation of neuron i in the second layer , we can write φ2 , x ( w 1 i ) = f2 ( 〈w1i , φ1 , x〉Φ1 ) with the layer non-linearity f2 applied elementwise and w1i the weights connecting the first layer to the i-th neuron in the second layer . In order for the second layer activations φ2 , x to be well-defined , φ2 , x ( w 1 i ) needs to be well-defined for every i , i.e . the inner products 〈w1i , φ1 , x〉Φ1 need to be welldefined for every i . In order for this to be the case , the weights w1i have to lie in the same space as φ1 , x for every i . Since φ1 , x ∈ Φ1 , we need to construct a sampling distribution over Φ1 for the weights connecting the first and second layers . As Φ1 is a space of functions defined over Rd , a natural choice for the sampling distribution of w1· is a GP . For constructing infinite-width networks with at most two layers , Hazan & Jaakkola ( 2015 ) endowed this GP with a general covariance function . However , as they did not take into account the structure of the underlying space Φ1 , this GP will , in general , not define a sampling distribution over Φ1 , thus yielding a network which is ill-defined as the weights will not sampled from the appropriate space . Analogously to the previous two layers , in order to construct a third infinite-width layer , we need to define the weight distribution over the space of functions of functions over Rd that is Φ2 = Span ( { φ2 , x| x ∈ X } ) as to ensure that taking inner products with φ2 , x is well-defined . Unfortunately , until now it was not clear how to construct this sampling distribution . In summary , the main challenge in constructing infinite-width networks is defining appropriate sampling distributions for the weights over increasingly more complex function spaces that are induced by the hidden layer activations to ensure that the network is well-defined . In particular , in order to construct layer l + 1 of infinite-width , we need to construct a distribution over the function space of activations at layer l , thus making the inner products between the weights and the l-th layer activations well-defined . Constructing the appropriate weight distributions is a highly non-trivial undertaking for networks with more than two layers of infinite-width as it is not clear how to go about constructing distributions over spaces of functions of functions .

Summary: In this paper, the authors propose deep neural networks of infinite width. The primary challenge in such networks is defining a distribution over the weights connecting two layers of infinite width. The authors tackle this by using Gaussian Processes for these distributions with the covariance functions defined in a canonical manner. Inspired by these networks, the authors propose weight initialization schemes for finite width networks.