Trajectory growth through random deep ReLU networks

1 INTRODUCTION . Deep neural networks continue to set new benchmarks for machine learning accuracy across a wide range of tasks , and are the basis for many algorithms we use routinely and on a daily basis . One fundamental set of theoretical questions concerning deep networks relates to their expressivity . There remain different approaches to understanding and quantifying neural network expressivity . Some results take a classical approximation theory approach , focusing on the relationship between the architecture of the network and the classes of functions it can accurately approximate ( Lu et al . ( 2017 ) ; Cybenko ( 1992 ) ; Hornik et al . ( 1989 ) ) . Another more recent approach has been to apply persistent homology to characterise expressivity ( Guss & Salakhutdinov ( 2018 ) ) , while Poole et al . ( 2016 ) focus on global curvature , and the ability of deep networks to disentangle manifolds . Other works concentrate specifically on networks with piecewise linear activation functions , using the number of linear regions ( Montufar et al . ( 2014 ) ) or the volume of the boundaries between linear regions ( Hanin & Rolnick ( 2019 ) ) in input space . In 2017 , Raghu et al . ( 2017 ) proposed trajectory length as a measure of expressivity ; in particular , they consider the expected change in length of a one-dimensional trajectory as it is passed through Gaussian random neural networks ( see Figure 1 for an illustration ) . Their primary theoretical result was that , in expectation , the length of a one-dimensional trajectory which is passed through a fully-connected , Gaussian network is lower bounded by a factor that is exponential with depth , but not with width . One-dimensional trajectories and their evolution through deep networks are also of interest in their own right because they constitute simple data manifolds . Firstly , we commonly assume that the real data which we aim to correctly classify or predict with a deep network lie on one or more manifolds , and thus design a network to perform appropriately on such a manifold . Secondly , researchers are beginning to consider whether the output ( manifolds ) of generator networks could be a good model for real word data manifolds , for example , as priors for a variety of inverse problems ( Manoel et al . ( 2017 ) ; Huang et al . ( 2018 ) ) . Both of these hypotheses motivate an understanding of how manifolds are acted upon by deep networks . Our results in this paper pertain specifically to the ‘ trajectory length ’ measure of expressivity . We produce a simpler proof than in the pioneering work of Raghu et al . ( 2017 ) , which also generalises their results , deriving similar lower bounds for a broader class of random deep neural networks . Theoretical work of this nature is important because it allows for more straightforward transfer and adaptation of prior theoretical results to new contexts of interest . For example , there is a current surge in research around low-memory networks , training sparse networks , and network pruning . Sparsely connected networks have shown the capacity to retain very high test accuracy ( Frankle & Carbin ( 2019 ) ; Han et al . ( 2015 ) ) , increased robustness ( Ahmad & Scheinkman ( 2019 ) ; Aghasi et al . ( 2017 ) ) , with much smaller memory footprints , and less power consumption ( Yang et al . ( 2019 ) ) . The approach we take in this work enables us to extend results from dense random networks to sparse ones . It also allows us to consider the other weight distributions of sparse-Gaussian , sparse-uniform and sparse-discrete networks ( see Definitions 2 - 4 ) . More specifically we make the following contributions : Contributions : 1 . We provide an alternative , simpler method for lower bounding expected trajectory growth through random networks , for a more general class of weights distributions ( Theorem 2 ) . 2 . We illustrate this approach by deriving bounds for sparse-Gaussian , sparse-uniform , and sparse-discrete random nets . We prove that trajectory growth can be exponential in depth with these distributions , with the sparsity appearing in the base of the exponential ( Corollaries 1 - 3 ) . 3 . We observe that the expected length growth factor is strikingly similar across the aforementioned three distributions . This suggests a universality of the expected growth in length for iid centered distributions determined only by the variance and sparsity ( Figure 3 ) . 1.1 NOTATION . We consider feedforward ReLU deep neural networks . We denote a the d-th post-activation layer as z ( d ) , and the subsequent pre-activation layer as h ( d ) , such that h ( d ) = W ( d ) z ( d ) + b ( d ) , z ( d+1 ) = φ ( h ( d ) ) , where φ ( x ) : = max ( x , 0 ) is applied elementwise . We denote x = z ( 0 ) . We use fNN ( x ; P , Q ) to denote a random feedforward deep neural network which takes as input the vector x , and is parameterised by random weight matrices W ( d ) with entries sampled iid from the distribution P , and bias vectors b ( d ) with entries drawn iid from distribution Q . Definition 1 . A random sparse network with sparsity parameter α , denoted fNN ( x ; α , P , Q ) , is a random feedforward network in which all weights are sampled from a mixture distribution of the form wij ∼ αP + ( 1− α ) δ , where δ is the delta distribution at 0 , and P is some other distribution . In other words , weights are 0 with probability 1− α , and sampled from P with probability α. Biases are drawn iid from Q . Definition 2 . A sparse-Gaussian network is a random sparse network fNN ( x ; α , P , Q ) , where P = N ( 0 , σ2w ) and Q = N ( 0 , σ2b ) . Definition 3 . A sparse-uniform network is a random sparse network fNN ( x ; α , P , Q ) , where P = U ( −Cw , Cw ) and Q = U ( −Cb , Cb ) . Definition 4 . A sparse-discrete network is a random sparse network fNN ( x ; α , P , Q ) , where P is a uniform distribution over a finite , discrete , symmetric setW , with cardinality |W| = Nw , and Q is a uniform distribution over a finite , discrete , symmetric set B , with cardinality |B| = Nb . For a weight matrix W in a random sparse network , with wi denoting the ith row , we define wPi as the vector containing only the P-distributed entries of wi . We define a trajectory x ( t ) in input space as a curve between two points , say x0 and x1 , parameterized by a scalar t ∈ [ 0 , 1 ] , with x ( 0 ) = x0 and x ( 1 ) = x1 , and we define z ( d ) ( x ( t ) ) = z ( d ) ( t ) to be the image of the trajectory in layer d of the network . The trajectory length l ( x ( t ) ) is given by the standard arc length , ∫ t ∣∣∣∣∣∣∣∣dx ( t ) dt ∣∣∣∣∣∣∣∣dt . As in the work by Raghu et al . ( 2017 ) , this paper considers trajectories with x ( t+ dt ) having a nontrivial component perpendicular to x ( t ) for all t , dt . Finally , we say a probability density or mass function fX ( x ) is even if fX ( −x ) = fX ( x ) for all random vectors x in the sample space . 2 EXPECTED TRAJECTORY GROWTH THROUGH RANDOM NETWORKS . Raghu et al . ( 2017 ) considered ReLU and hard-tanh Gaussian networks with the standard deviation scaled by 1/ √ k. Their result with respect to ReLU networks is captured in the following theorem . Theorem 1 ( Raghu et al . ( 2017 ) ) . Let fNN ( x ; N ( 0 , σ2w/k ) , N ( 0 , σ2b ) ) be a random Gaussian deep ReLU neural network with layers of width k , then E [ l ( z ( d ) ( t ) ) ] ≥ O ( σw √ k√ k + 1 ) d · l ( x ( t ) ) , for x ( t ) a 1-dimensional trajectory in input space . There are , however , other network weight distributions which may be of interest . For example , the expressivity and generative power of sparse networks are of particular interest in the current moment , given the current interest in low-memory and low-energy networks , training sparse networks , and network pruning ( Frankle & Carbin ( 2019 ) ; Han et al . ( 2015 ) ; Yang et al . ( 2019 ) ) . We prove that even for sparse random networks , trajectory growth can remain exponential in depth given sufficiently large initialisation scale σw . Scaling σw by 1/ √ k can yield a width-independent lower bound on this growth . Moreover , a sufficiently high sparsity fraction ( 1 − α ) results in a lower bound which , instead of growing exponentially , shrinks exponentially to zero . This is captured by the following result . Corollary 1 ( Trajectory growth in deep sparse-Gaussian random networks ) . Let fNN ( x ; α , N ( 0 , σ2w ) , N ( 0 , σ2b ) ) be a sparse-Gaussian , feedforward ReLU network as defined in Section 1.1 , with layers of width k. Then E [ l ( z ( d ) ( t ) ) ] ≥ ( ασw √ k√ 2π ) d · l ( x ( t ) ) , ( 1 ) for x ( t ) a 1-dimensional trajectory in input space . Corollary 1 with α = 1 and σw replaced by σw/ √ k recovers a bound which is very similar to the prior bound by Raghu et al . ( 2017 ) in Theorem 1 . Beyond Gaussian weights , we consider other distributions commonly used for initialising and analysing deep networks . Uniform distributions , for example , still constitute the default initialisations of linear network layers in both Pytorch and Tensorflow ( uniform according to U ( −1/ √ k , 1/ √ k ) in the case of Pytorch , and uniform according to U ( −6/ √ kin + kout , 6/ √ kin + kout ) – a.k.a the Glorot/Xavier uniform initialization ( Glorot & Bengio ( 2010 ) ) – in the case of Tensorflow ) . We prove an analogous lower bound for uniformly distributed weights . Corollary 2 ( Trajectory growth in deep sparse-uniform random networks ) . Let fNN ( x ; α , U ( −Cw , Cw ) , U ( −Cb , Cb ) ) be a sparse-Uniform , feedforward ReLU network as defined in Section 1.1 , with layers of width k. Then E [ l ( z ( d ) ( t ) ) ] ≥ ( αCw √ k 4 √ 2 ) d · l ( x ( t ) ) , ( 2 ) for x ( t ) a 1-dimensional trajectory in input space . Another research direction which has gathered some momentum in recent years are quantized or discrete-valued deep neural networks ( Li et al . ( 2017 ) ; Hubara et al . ( 2016 ; 2017 ) ) , including recent work using integer valued weights ( Wu et al . ( 2018 ) ) . This motivates consideration of discrete weight distributions , in addition to continuous ones . As an example of such , we prove a similar lower bound for networks with weights and biases uniformly sampled from finite , symmetric , discrete sets . Corollary 3 ( Trajectory growth in deep sparse-discrete random networks ) . Let fNN ( x ; α , P , Q ) be a sparse-discrete random feedforward ReLU network as defined in Section 1.1 , and layers of width k. Then E [ l ( z ( d ) ( t ) ) ] ≥ ( α √ k 2 √ 2 · ∑ w∈W |w| Nw ) d · l ( x ( t ) ) ( 3 ) for x ( t ) a 1-dimensional trajectory in input space . In all cases these lower bounds show how to choose the combination of σw and α to guarantee ( or not ) exponential growth in trajectory length in expectation at initialisation . The main idea behind the derivation of these results is to consider how the length of a small piece of a trajectory ( some ‖dz ( d ) ‖ ) grows from one layer to the next ( ‖dz ( d+1 ) ‖ = ‖φ ( hd ( t + dt ) ) − φ ( h ( d ) ( t ) ‖ ) . In the context of random feedforward networks , we can consider piecewise linear activation functions as restrictions of dh ( d ) to a particular support set which is statistically dependent on h ( d ) . This approach was developed by Raghu et al . ( 2017 ) . The key to our proof is providing a more direct and more generally applicable way of accounting for this dependence than originally provided by Raghu et al . ( 2017 ) . Specifically , our approach lets us derive the following , more general result , from which Corollaries 1 , 2 , and 3 follow easily . Theorem 2 ( Trajectory growth in deep random sparse networks ) . Let fNN ( x ; α , P , Q ) be a random sparse network as defined in Section 1.1 , with layers of width k. Let P and Q be such that the joint distribution over a vector of independent elements from both distributions is even . If E [ |u > ŵPi | ] ≥M‖u‖ for any constant vector u , for all i , then E [ l ( z ( d ) ( t ) ) ] ≥ ( αM √ k 2 ) d · l ( x ( t ) ) ( 4 ) for x ( t ) a 1-dimensional trajectory in input space . Remark . It is trivial to amend this result for networks where the width , distribution , and sparsity varies layer by layer , in which case the lower bound ( 4 ) is replaced by d∏ j=i ( αjMj √ kj 2 ) · l ( x ( t ) ) Moreover , the bounds from Theorem 2 and Corollaries 1 - 3 hold true in the 0 bias case as well .

Summary: This submission proposes an alternative way to lower bound the trajectory growth through random networks. It generalizes to a variety of weights distributions. For example, the authors showcase their approach on sparse-Gaussian, sparse-uniform, and sparse-discrete-valued random nets and prove that trajectory growth can be exponential in depth with these distributions, with the sparsity appearing in the base of the exponential.