Trajectory growth through random deep ReLU networks

1 INTRODUCTION . Deep neural networks continue to set new benchmarks for machine learning accuracy across a wide range of tasks , and are the basis for many algorithms we use routinely and on a daily basis . One fundamental set of theoretical questions concerning deep networks relates to their expressivity . There remain different approaches to understanding and quantifying neural network expressivity . Some results take a classical approximation theory approach , focusing on the relationship between the architecture of the network and the classes of functions it can accurately approximate ( Lu et al . ( 2017 ) ; Cybenko ( 1992 ) ; Hornik et al . ( 1989 ) ) . Another more recent approach has been to apply persistent homology to characterise expressivity ( Guss & Salakhutdinov ( 2018 ) ) , while Poole et al . ( 2016 ) focus on global curvature , and the ability of deep networks to disentangle manifolds . Other works concentrate specifically on networks with piecewise linear activation functions , using the number of linear regions ( Montufar et al . ( 2014 ) ) or the volume of the boundaries between linear regions ( Hanin & Rolnick ( 2019 ) ) in input space . In 2017 , Raghu et al . ( 2017 ) proposed trajectory length as a measure of expressivity ; in particular , they consider the expected change in length of a one-dimensional trajectory as it is passed through Gaussian random neural networks ( see Figure 1 for an illustration ) . Their primary theoretical result was that , in expectation , the length of a one-dimensional trajectory which is passed through a fully-connected , Gaussian network is lower bounded by a factor that is exponential with depth , but not with width . One-dimensional trajectories and their evolution through deep networks are also of interest in their own right because they constitute simple data manifolds . Firstly , we commonly assume that the real data which we aim to correctly classify or predict with a deep network lie on one or more manifolds , and thus design a network to perform appropriately on such a manifold . Secondly , researchers are beginning to consider whether the output ( manifolds ) of generator networks could be a good model for real word data manifolds , for example , as priors for a variety of inverse problems ( Manoel et al . ( 2017 ) ; Huang et al . ( 2018 ) ) . Both of these hypotheses motivate an understanding of how manifolds are acted upon by deep networks . Our results in this paper pertain specifically to the ‘ trajectory length ’ measure of expressivity . We produce a simpler proof than in the pioneering work of Raghu et al . ( 2017 ) , which also generalises their results , deriving similar lower bounds for a broader class of random deep neural networks . Theoretical work of this nature is important because it allows for more straightforward transfer and adaptation of prior theoretical results to new contexts of interest . For example , there is a current surge in research around low-memory networks , training sparse networks , and network pruning . Sparsely connected networks have shown the capacity to retain very high test accuracy ( Frankle & Carbin ( 2019 ) ; Han et al . ( 2015 ) ) , increased robustness ( Ahmad & Scheinkman ( 2019 ) ; Aghasi et al . ( 2017 ) ) , with much smaller memory footprints , and less power consumption ( Yang et al . ( 2019 ) ) . The approach we take in this work enables us to extend results from dense random networks to sparse ones . It also allows us to consider the other weight distributions of sparse-Gaussian , sparse-uniform and sparse-discrete networks ( see Definitions 2 - 4 ) . More specifically we make the following contributions : Contributions : 1 . We provide an alternative , simpler method for lower bounding expected trajectory growth through random networks , for a more general class of weights distributions ( Theorem 2 ) . 2 . We illustrate this approach by deriving bounds for sparse-Gaussian , sparse-uniform , and sparse-discrete random nets . We prove that trajectory growth can be exponential in depth with these distributions , with the sparsity appearing in the base of the exponential ( Corollaries 1 - 3 ) . 3 . We observe that the expected length growth factor is strikingly similar across the aforementioned three distributions . This suggests a universality of the expected growth in length for iid centered distributions determined only by the variance and sparsity ( Figure 3 ) . 1.1 NOTATION . We consider feedforward ReLU deep neural networks . We denote a the d-th post-activation layer as z ( d ) , and the subsequent pre-activation layer as h ( d ) , such that h ( d ) = W ( d ) z ( d ) + b ( d ) , z ( d+1 ) = φ ( h ( d ) ) , where φ ( x ) : = max ( x , 0 ) is applied elementwise . We denote x = z ( 0 ) . We use fNN ( x ; P , Q ) to denote a random feedforward deep neural network which takes as input the vector x , and is parameterised by random weight matrices W ( d ) with entries sampled iid from the distribution P , and bias vectors b ( d ) with entries drawn iid from distribution Q . Definition 1 . A random sparse network with sparsity parameter α , denoted fNN ( x ; α , P , Q ) , is a random feedforward network in which all weights are sampled from a mixture distribution of the form wij ∼ αP + ( 1− α ) δ , where δ is the delta distribution at 0 , and P is some other distribution . In other words , weights are 0 with probability 1− α , and sampled from P with probability α. Biases are drawn iid from Q . Definition 2 . A sparse-Gaussian network is a random sparse network fNN ( x ; α , P , Q ) , where P = N ( 0 , σ2w ) and Q = N ( 0 , σ2b ) . Definition 3 . A sparse-uniform network is a random sparse network fNN ( x ; α , P , Q ) , where P = U ( −Cw , Cw ) and Q = U ( −Cb , Cb ) . Definition 4 . A sparse-discrete network is a random sparse network fNN ( x ; α , P , Q ) , where P is a uniform distribution over a finite , discrete , symmetric setW , with cardinality |W| = Nw , and Q is a uniform distribution over a finite , discrete , symmetric set B , with cardinality |B| = Nb . For a weight matrix W in a random sparse network , with wi denoting the ith row , we define wPi as the vector containing only the P-distributed entries of wi . We define a trajectory x ( t ) in input space as a curve between two points , say x0 and x1 , parameterized by a scalar t ∈ [ 0 , 1 ] , with x ( 0 ) = x0 and x ( 1 ) = x1 , and we define z ( d ) ( x ( t ) ) = z ( d ) ( t ) to be the image of the trajectory in layer d of the network . The trajectory length l ( x ( t ) ) is given by the standard arc length , ∫ t ∣∣∣∣∣∣∣∣dx ( t ) dt ∣∣∣∣∣∣∣∣dt . As in the work by Raghu et al . ( 2017 ) , this paper considers trajectories with x ( t+ dt ) having a nontrivial component perpendicular to x ( t ) for all t , dt . Finally , we say a probability density or mass function fX ( x ) is even if fX ( −x ) = fX ( x ) for all random vectors x in the sample space . 2 EXPECTED TRAJECTORY GROWTH THROUGH RANDOM NETWORKS . Raghu et al . ( 2017 ) considered ReLU and hard-tanh Gaussian networks with the standard deviation scaled by 1/ √ k. Their result with respect to ReLU networks is captured in the following theorem . Theorem 1 ( Raghu et al . ( 2017 ) ) . Let fNN ( x ; N ( 0 , σ2w/k ) , N ( 0 , σ2b ) ) be a random Gaussian deep ReLU neural network with layers of width k , then E [ l ( z ( d ) ( t ) ) ] ≥ O ( σw √ k√ k + 1 ) d · l ( x ( t ) ) , for x ( t ) a 1-dimensional trajectory in input space . There are , however , other network weight distributions which may be of interest . For example , the expressivity and generative power of sparse networks are of particular interest in the current moment , given the current interest in low-memory and low-energy networks , training sparse networks , and network pruning ( Frankle & Carbin ( 2019 ) ; Han et al . ( 2015 ) ; Yang et al . ( 2019 ) ) . We prove that even for sparse random networks , trajectory growth can remain exponential in depth given sufficiently large initialisation scale σw . Scaling σw by 1/ √ k can yield a width-independent lower bound on this growth . Moreover , a sufficiently high sparsity fraction ( 1 − α ) results in a lower bound which , instead of growing exponentially , shrinks exponentially to zero . This is captured by the following result . Corollary 1 ( Trajectory growth in deep sparse-Gaussian random networks ) . Let fNN ( x ; α , N ( 0 , σ2w ) , N ( 0 , σ2b ) ) be a sparse-Gaussian , feedforward ReLU network as defined in Section 1.1 , with layers of width k. Then E [ l ( z ( d ) ( t ) ) ] ≥ ( ασw √ k√ 2π ) d · l ( x ( t ) ) , ( 1 ) for x ( t ) a 1-dimensional trajectory in input space . Corollary 1 with α = 1 and σw replaced by σw/ √ k recovers a bound which is very similar to the prior bound by Raghu et al . ( 2017 ) in Theorem 1 . Beyond Gaussian weights , we consider other distributions commonly used for initialising and analysing deep networks . Uniform distributions , for example , still constitute the default initialisations of linear network layers in both Pytorch and Tensorflow ( uniform according to U ( −1/ √ k , 1/ √ k ) in the case of Pytorch , and uniform according to U ( −6/ √ kin + kout , 6/ √ kin + kout ) – a.k.a the Glorot/Xavier uniform initialization ( Glorot & Bengio ( 2010 ) ) – in the case of Tensorflow ) . We prove an analogous lower bound for uniformly distributed weights . Corollary 2 ( Trajectory growth in deep sparse-uniform random networks ) . Let fNN ( x ; α , U ( −Cw , Cw ) , U ( −Cb , Cb ) ) be a sparse-Uniform , feedforward ReLU network as defined in Section 1.1 , with layers of width k. Then E [ l ( z ( d ) ( t ) ) ] ≥ ( αCw √ k 4 √ 2 ) d · l ( x ( t ) ) , ( 2 ) for x ( t ) a 1-dimensional trajectory in input space . Another research direction which has gathered some momentum in recent years are quantized or discrete-valued deep neural networks ( Li et al . ( 2017 ) ; Hubara et al . ( 2016 ; 2017 ) ) , including recent work using integer valued weights ( Wu et al . ( 2018 ) ) . This motivates consideration of discrete weight distributions , in addition to continuous ones . As an example of such , we prove a similar lower bound for networks with weights and biases uniformly sampled from finite , symmetric , discrete sets . Corollary 3 ( Trajectory growth in deep sparse-discrete random networks ) . Let fNN ( x ; α , P , Q ) be a sparse-discrete random feedforward ReLU network as defined in Section 1.1 , and layers of width k. Then E [ l ( z ( d ) ( t ) ) ] ≥ ( α √ k 2 √ 2 · ∑ w∈W |w| Nw ) d · l ( x ( t ) ) ( 3 ) for x ( t ) a 1-dimensional trajectory in input space . In all cases these lower bounds show how to choose the combination of σw and α to guarantee ( or not ) exponential growth in trajectory length in expectation at initialisation . The main idea behind the derivation of these results is to consider how the length of a small piece of a trajectory ( some ‖dz ( d ) ‖ ) grows from one layer to the next ( ‖dz ( d+1 ) ‖ = ‖φ ( hd ( t + dt ) ) − φ ( h ( d ) ( t ) ‖ ) . In the context of random feedforward networks , we can consider piecewise linear activation functions as restrictions of dh ( d ) to a particular support set which is statistically dependent on h ( d ) . This approach was developed by Raghu et al . ( 2017 ) . The key to our proof is providing a more direct and more generally applicable way of accounting for this dependence than originally provided by Raghu et al . ( 2017 ) . Specifically , our approach lets us derive the following , more general result , from which Corollaries 1 , 2 , and 3 follow easily . Theorem 2 ( Trajectory growth in deep random sparse networks ) . Let fNN ( x ; α , P , Q ) be a random sparse network as defined in Section 1.1 , with layers of width k. Let P and Q be such that the joint distribution over a vector of independent elements from both distributions is even . If E [ |u > ŵPi | ] ≥M‖u‖ for any constant vector u , for all i , then E [ l ( z ( d ) ( t ) ) ] ≥ ( αM √ k 2 ) d · l ( x ( t ) ) ( 4 ) for x ( t ) a 1-dimensional trajectory in input space . Remark . It is trivial to amend this result for networks where the width , distribution , and sparsity varies layer by layer , in which case the lower bound ( 4 ) is replaced by d∏ j=i ( αjMj √ kj 2 ) · l ( x ( t ) ) Moreover , the bounds from Theorem 2 and Corollaries 1 - 3 hold true in the 0 bias case as well .

Summary: The authors examine trajectory growth of deep ReLU neural networks whose weights come from a (random) “sparse-Gaussian”, “sparse-uniform”, and “sparse-discrete” distribution. They give definitions of these distributions in the paper. They do this by extending the proof of Raghu (2017) so that it can handle more general distributions than the standard Gaussian. They also provide some numerical experiments verifying their theories.