The Benefits of Over-parameterization at Initialization in Deep ReLU Networks

1 INTRODUCTION . Deep rectifier ( ReLU ) networks are popular in deep learning due to their ease of training and stateof-the-art generalization . This success of deep rectifier networks can be partly attributed to good initialization strategies ( for example Glorot & Bengio ( 2010 ) ; He et al . ( 2015 ) ) . Essentially , good parameter initializations guarantee that there is no exploding or vanishing of information across hidden layers . These properties help gradient descent based optimization methods in navigating the complex non-linear loss landscape of deep networks by initializing them at a good starting point where training can begin . Such favorable properties promised by these initialization strategies are ( in most cases ) shown to hold true in asymptotic settings where the network width tends to infinity and/or under strict assumptions made about the distribution of the input data . A detailed account of these existing papers and a contrast between these papers and our work is discussed in section 2 . Our paper relaxes the aforementioned assumptions made in previous papers . Further , we show novel properties that hold for deep ReLU networks at initialization when using the He initialization scheme ( He et al. , 2015 ) . Specifically , we show that deep ReLU networks obey the following properties in the forward ( Eq . 1 ) back backward ( Eq . 2 ) pass ( see section 3 for notations ) , ‖hl‖2 ≈ ‖x‖2 ∀l ∈ { 1 , 2 , · · · , L } ( 1 ) ‖∂ ` ( fθ ( x ) , y ) ∂Wl ‖F ≈ ‖δ ( x , y ) ‖2 · ‖x‖2 ∀l ∈ { 1 , 2 , · · · , L } ( 2 ) We refer to the above properties as as the the activation norm equality and the gradient norm equality property . Further , we derive a finite lower bound on the width of the hidden layers for which the above results hold ( i.e. , the network needs to be sufficiently over-parameterized ) in contrast to a number of previous papers that assume infinitely wide layers . We do not make any assumption on the data distribution as done in a number of previous papers that study initialization . Further , our results hold for an infinite stream of data for the activation norm equality property and for any finite dataset in the backward pass . Thus we have relaxed a number of assumptions made in previous research work that focus on deriving initialization strategies for deep ReLU networks . Our results showing the connection between activation norm and input norm ( and similarly the property for gradients ) for deep ReLU networks can be utilized in further research studies . 2 RELATION WITH EXISTING WORK . The seminal work of Glorot & Bengio ( 2010 ) studied for the first time a principled way to initialize deep networks to avoid exploding/vanishing gradient problem ( EVGP ) . Their analysis however is done for deep linear networks . The analysis by He et al . ( 2015 ) follows the derivation strategy of Glorot & Bengio ( 2010 ) except they tailor their derivation for deep ReLU networks . However , both these papers make a strong assumption that the dimensions of the input are statistically independent and that the network width is infinite . Our results do not make these assumptions . Saxe et al . ( 2013 ) introduce the notion of dynamical isometry which is achieved when all the singular values of the input-output Jacobian of the network is 1 . They show that deep linear networks achieve dynamical isometry when initialized using orthogonal weights and this property allows fast learning in such networks . Poole et al . ( 2016 ) study how the norm of hidden activations evolve when propagating an input through the network . Pennington et al . ( 2017 ; 2018 ) study the exploding and vanishing gradient problem in deep ReLU networks using tools from free probability theory . Under the assumption of an infinitely wide network , they show that the average squared singular value of the input-output Jacobian for deep ReLU network is 1 when initialized appropriately . Our paper on the other hand shows that deep ReLU networks are norm preserving maps at appropriate initialization . Further , we show there exists a finite lower bound on the width of the network for which the Frobenius norm of the hidden layer-output Jacobian ( equivalently the sum of its squared singular values ) are equal across all hidden layers . Hanin & Rolnick ( 2018 ) show that for a fixed input , the variance of the squared norm of hidden layer activations are bounded from above and below for deep ReLU networks to be near the squared norm of the input such that the bound depends on the sum of reciprocal of layer widths of the network . Our paper shows a similar result in a PAC bound sense but as an important difference , we show that these results hold even for an infinite stream of data by making the bound depend on the dimensionality of the input . Hanin ( 2018 ) show that sufficiently wide deep ReLU networks with appropriately initialized weights prevent EVGP in the sense that the fluctuation between the elements of the input-output Jacobian matrix of the network is small . This avoids EVGP because a large fluctuation between the elements of the input-output Jacobian implies a large variation in its singular values . Our paper shows that sufficiently wide deep ReLU networks avoid EVGP in the sense that the norm of the gradient for the weights of each layer is roughly equal to a fixed quantity that depends on the input and target . Over-parameterization in deep networks has previously been shown to have advantages . Neyshabur et al . ( 2014 ) ; Arpit et al . ( 2017 ) show empirically that wider networks train faster ( number of epochs ) and have better generalization performance . From a theoretical view point , Neyshabur et al . ( 2018 ) derive a generalization bound for a two layer ReLU network where they show that a wider network has a lower complexity . Lee et al . ( 2017 ) show that infinitely wide deep networks act as a Gaussian process . Arora et al . ( 2018 ) show that over-parameterization in deep linear networks acts as a conditioning on the gradient leading to faster convergence , although in this case overparameterization in terms of depth is studied . Our analysis complements this line of work by showing another advantage of over-parameterization in deep ReLU networks . 3 THEORETICAL RESULTS . Let D = { xi , yi } Ni=1 be N training sample pairs of inputs vectors xi ∈ Rno and target vectors yKi where xi ’ s are sampled from a distribution with support X . Define a L layer deep ReLU network fθ ( x ) = h L with the lth hidden layer ’ s activation given by , hl : = ReLU ( al ) al : = Wlhl−1 + bl l ∈ { 1 , 2 , · · ·L } ( 3 ) where hl ∈ Rnl are the hidden activations , ho is the input to the network and can be one of the input vectors xi , Wl ∈ Rnl×nl−1 are the weight matrices , b ∈ Rnl are the bias vectors which are initialized as 0s , al are the pre-activations and θ = { ( Wl , bl ) } Ll=1 . Define a loss on the deep network function for any given training data sample ( x , y ) as , ` ( fθ ( x ) , y ) ( 4 ) where ` ( . ) is any desired loss function . For instance , ` ( . ) can be log loss for a classification problem , in which case fθ ( x ) is transformed using a weight matrix to have dimensions equal to the number of classes and the softmax activation is applied to yield class probabilities ( i.e. , a logistic regression like model on top of fθ ( x ) ) . However for our purpose , we do not need to restrict ` ( . ) to a specific choice , we only need it to be differentiable . We will make use of the notation , δ ( x , y ) : = ∂ ` ( fθ ( x ) , y ) ∂aL ( 5 ) We organize our theoretical results as follows . We first derive the activation norm equality property for finite datasets and then extend these results to infinite dataset setting in section 3.1 . We then derive the gradient norm equality property for finite datasets in section 3.2 . All formal proofs , if not shown in the main text , are available in the appendix . 3.1 ACTIVATION NORM EQUALITY . Consider an L layer deep ReLU network and data x ∈ X . We show in this section that the norm of hidden layer activation of any layer is roughly equal to the norm of the input at initialization for all x ∈ X if the network weights are initialized appropriately and the network width is sufficiently large but finite . Specifically we show ∀l ∈ [ L ] and x ∈ X , ‖hl‖2 ≈ ‖x‖2 ( 6 ) To achieve this goal , we start with a very simple result– in expectation , ReLU transformation in each layer preserves the norm of its corresponding input if the weights are sampled appropriately . Evaluating this expectation also helps determining the scale of the random initialization that leads to norm preservation . Lemma 1 Let v = ReLU ( Ru ) , where u ∈ Rn , R ∈ Rm×n . If Rij i.i.d.∼ N ( 0 , 2m ) , then for any fixed vector u , E [ ‖v‖2 ] = ‖u‖2 . The proof for the above lemma involves simply computing the expectation analytically by exploting the fact that each dimension of the vector u is a weighted sum of Gaussian random variables . The above result thus shows that for each layer , initializing its weights from an i.i.d . Gaussian distribution with 0 mean and 2/fan-out variance ( viz . He initialization ( He et al. , 2015 ) ) preserves the norm of its input in expectation . We now derive a lower bound on the width of a ReLU layer so that it can preserve the norm of the input for a single fixed input with error margin . Lemma 2 Let v = ReLU ( Ru ) , where u ∈ Rn , R ∈ Rm×n . If Rij i.i.d.∼ N ( 0 , 2m ) , and ∈ [ 0 , 1 ) , then for any fixed vector u , Pr ( |‖v‖2 − ‖u‖2| ≤ ‖u‖2 ) ≥ 1− 2 exp ( −m ( 4 + log 2 1 + √ 1 + ) ) ( 7 ) The proof of this lemma involves a direct application of the Chernoff bounding technique . Now we use the above lemma to show that the norm of hidden activations equal the norm of inputs within a specified margin and for a finite size dataset for a deep ReLU network . Theorem 1 Let D be a fixed dataset with N samples and define a L layer ReLU network fθ ( . ) as shown in Eq . 3 such that each weight matrix Wl ∈ Rnl×nl−1 has its elements sampled as W lij i.i.d.∼ N ( 0 , 2nl ) and biases b l are set to zeros . Then for any sample ( x , y ) ∈ D and ∈ [ 0 , 1 ) , we have that , Pr ( ( 1− ) L‖x‖2 ≤ ‖fθ ( x ) ‖2 ≤ ( 1 + ) L‖x‖2 ) ≥ 1− L∑ l′=1 2N exp ( −nl′ ( 4 + log 2 1 + √ 1 + ) ) ( 8 ) While the statement of the above theorem only talks about the norm of the final output of the network , it equally applies to any hidden layer l as well since the theorem can be applied equivalently to a l layer network . Having proved the activation norm equality property in the finite dataset setting above , we now turn our attention to the case of infinite dataset case . To do so , we first prove a non-trivial result where we use lemma 2 to show how a lower bound on the width of an individual ReLU layer can be computed such that this layer preserves the norm of an infinite stream of inputs . Lemma 3 Let X be a d ≤ n dimensional subspace of Rn and R ∈ Rm×n . If Rij i.i.d.∼ N ( 0 , 2m ) , ∈ [ 0 , 1 ) , and , m ≥ 1 /12− log ( 0.5 ( 1 + √ 1 + /3 ) ) · ( d log 2 ∆ + log 4 δ ) ( 9 ) then with probability at least 1− δ , ( 1− ) ‖u‖2 ≤ ‖ReLU ( Ru ) ‖2 ≤ ( 1 + ) ‖u‖2 ∀u ∈ X ( 10 ) where ∆ : = min { 3 √ d , √ √ 3d } . Proof Sketch : The core idea behind the proof is inspired by lemma 10 of Sarlos ( 2006 ) . Without any loss of generality , we will show the norm preserving property for any unit vector u in the d dimensional subspace X of Rn . This is because for any arbitrary length vector u , ‖ReLU ( Ru ) ‖ = ‖u‖ · ‖ReLU ( Rû ) ‖ . The idea then is to define a grid of finite points over X on [ −1 , 1 ] d with interval size depending on , such that every unit vector û in X is close enough to one of the grid points . Then , if we choose the width of the layer to be large enough to approximately preserve the length of the finite number of grid points , we can guarantee that the length of any arbitrary unit vector approximately remains preserved as well within the derived margin of error . The formal proof can be found in the appendix . We now extend the above lemma to a deep ReLU network and show our main result for the forward pass that the norm of hidden activations equal the norm of input within some distortion margin , for an infinite stream of input data for a sufficiently large ( but finite ) width deep ReLU network . Theorem 2 Define a L layer ReLU network fθ ( . ) as shown in Eq . 3 such that each weight matrix Wl ∈ Rnl×nl−1 has its elements sampled as W lij i.i.d.∼ N ( 0 , 2nl ) and biases b l are set to zeros . Let X be a d ≤ n dimensional subspace of Rn . If W lij i.i.d.∼ N ( 0 , 2nl ) , ∈ [ 0 , 1 ) , and , nl ≥ 1 /12− log ( 0.5 ( 1 + √ 1 + /3 ) ) · ( d log 2 ∆ + log 4L δ ) ∀l ∈ [ L ] ( 11 ) then with probability at least 1− δ , ( 1− ) l‖x‖2 ≤ ‖hl‖2 ≤ ( 1 + ) l‖x‖2 ∀x ∈ X ∀l ∈ [ L ] ( 12 ) Proof : Since the input lies on a d dimensional subspace of Rn , we apply theorem 3 to the first layer and get the guarantee that the norm of all inputs on the d dimensional subspace are preserved by this layer . Next , we show that since each layer is a linear transform followed by pointwise ReLU non-linearity , and the input takes values in a set defined by the d dimensional subspace of Rn , the output of the first layer will take values in a set that is strictly a subset of a d dimensional subspace . To see this , let B ∈ Rn×d denote a matrix with orthonormal columns describing the basis of the subspace X on which the input lies , and let z ∈ Rd . Then we have that , x ∈ { Bz|z ∈ Rd } ( 13 ) The first layer transforms any input x as h1 = ReLU ( W0x ) ( 14 ) = ReLU ( W0Bz ) ( 15 ) Denote B′ = W0B . Then note that rank ( B′ ) ≤ d. Let S1 denote the set of values that h1 can take . Then we have that , S1 = { B′z|z ∈ Rd } ∩ Rn + 1 ( 16 ) where Rn+ denote the subset of Rn where all dimensions take non-negative values . This shows that h1 takes values in a set that is strictly a subset of a d dimensional subspace of Rn1 . Having proved this for first layer , we can recursively apply this strategy to all higher layers since the output of each layer lies on a subset of a subspace . Notice that while doing so , the lower bound on the width of each layer depends only on the subspace dimensionality d. Applying union bound over the result of theorem 3 for L layers proves the claim . We note that the lower bound on width derived above depends on two quantities– the depth of the network L , and the dimensionality of the subspace d on which the input lies . Specifically , the lower bound on the width becomes larger for larger input dimensionality d and larger network depth L irrespective of the number of data samples , meaning that a wider network is needed as the depth of the network and/or the intrinsic input dimensionality increases .

Summary: This paper studies initialization techniques for deep ReLU networks from a theoretical standpoint and derives finite layer width concentration bounds to show that with the He initialization scheme, deep ReLU networks preserve the norm of the input sample during a forward pass and the norm of the gradient with respect to the output during a backward pass. The concentration bounds also suggest lower bounds on the width of the ReLU layers. The authors verify their theory with experiments on synthetic data.