The Effect of Residual Architecture on the Per-Layer Gradient of Deep Networks

A critical part of the training process of neural networks takes place in the very first gradient steps post-initialization . In this work , we study the connection between the network ’ s architecture and initialization parameters , to the statistical properties of the gradient in random fully connected ReLU networks , through the study of the the per layer Jacobian in finite sized networks . We compare three types of architectures : vanilla networks , ResNets and DenseNets . In our analysis , we show that while the variance of Jacobian squared norm is exponential in depth for ResNets , and polynomial for DenseNets , there exists an initialization strategy for both , such that the norm is preserved through arbitrary depths , preventing exploding or decaying gradients in deep networks.We also show that the statistics of the per layer Jacobian norm is a function of the architecture and the layer ’ s size , but surprisingly , not the layer ’ s depth . 1 INTRODUCTION . Understanding the effect of different architectures on the ability to train deep networks has long been a major topic of research . A recent line of work has focused on the statistical properties of both the activations and the gradients of different architectures at the point of initialization , in order to better understand the effect of different architectural choices on the training dynamics . A well-known hurdle in the ability to train deep networks , is the effect of depth on the magnitude of the gradients . Specifically , the exploding and vanishing gradient phenomena have been observed in various deep architectures . they are considered a main culprit in the failure of training deep networks successfully . A flurry of architectural tricks and initialization techniques have been proposed , in order to mitigate the problem of exploding and vanishing gradients , ranging from residual architectures , different normalization techniques and orthogonal initializations . In this work , we perform a rigorous analysis of the norm , of both the activations and the gradients of each layer , in fully connected architectures at the point of initialization . Our setting is as follows : we assume a fixed input y0 to an L layer neural network with weights given by W , and intermediate output for any layer l denoted by yl . Denoting by wkij the weights ij of layer k , we are interested in the quantities : ‖yl‖2 , ‖Jk‖2 = ∑ ij ‖ ∂y L ∂wkij ‖2 ( 1 ) representing the norm of the activations of layer l and Jacobian with respect to layer k. We consider these quantities random variables at initialization , and , therefore , devote our analysis to deriving expressions for the mean and variance of both ‖yl‖2 and ‖Jk‖2 , over the random initializations of the weights . Note that the full gradient requires an additional loss term which is independent of the architecture , and so our attention is focused on Jk . While most initialization techniques are designed to maintain the squared length of intermediate activations through the layers , the multiplication of i.i.d random weights along input-to-output paths in increasingly deep networks might lead to unbounded higher order statistics . In the extreme case , while the mean of activation and gradients remain constant in deep layers , it would be improbable to achieve mean values for any specific realization of the weights , rendering training difficult . Specifically , given some loss function L , the norm of the gradient with respect to W k is given by : ‖∇WkL‖ = ‖ ∂L ∂yL Jk‖ ≤ ‖ ∂L ∂yL ‖‖Jk‖ ( 2 ) Large fluctuations to the Jacobian norm with respect to small perturbations of the weights will , therefore , cause wildly fluctuating gradients during training , hindering training dynamics . Our analysis makes use of a duality theorem , which captures an intuitive concept . In residual architectures , there are many alternative paths of varying lengths from input to output . The theorem states that when considering the moments of the gradient at a certain layer , one can change the architecture , so all connections which that skip that layer are removed . This is true , despite the activations of the two networks being very different . See Fig . 1 . Our contributions are as follows : ( i ) We rigorously study the effect of different architectures , namely vanilla , residual and densely connected architectures , on the mean and variance of ‖yl‖2 and ‖Jk‖2 in finite depth and width networks . ( ii ) From our forward-backward norm propagation duality theorem , we derive conditions that prevent exploding gradients in each of these architectures . ( iii ) We prove that for densely connected networks , the commonly used initialization of He et al . ( 2015 ) is enough to ensure the boundness of var ( ‖Jk‖2 ) and var ( ‖yl‖2 ) for arbitrarily deep networks , shedding new light on the apparent advantage of these architectures in practical applications . ( iv ) We prove that replacing the standard conv-ReLU blocks with blocks of the concatenated ReLU ( CR ) activation presented by Shang et al . ( 2016 ) , while maintaining the same parameter budget , significantly reduces the fluctuations of both the activations and gradients at initialization for any ReLU based architecture . We evaluate the applicability of this simple alteration on a large number of UCI datasets , and demonstrate the advantage for CR based networks over their standard counterparts . 2 RELATED WORK . The effect of architecture and depth on the gradients has been a topic of considerable interest recently , leading to several insightful research directions . A major line of work has been focused on the mean field approach , where the propagation of signals in random networks are analyzed in the regime of infinite width ( Yang & Schoenholz ( 2017 ) , Schoenholz et al . ( 2017 ) , Xiao et al . ( 2018 ) ) . In that case , the pre-activations in any layer are approximated by a Gaussian distribution with moments depending on the specific layer . For ReLU networks ( without residual connections ) , controlling the exponential explosion or decay of both the activations and gradients with depth is a simple matter of correctly scaling the initialization parameters . Indeed , it is shown in Schoenholz et al . ( 2017 ) that classical feed forward networks exhibit exponential behavior on average , when not initialized properly . For the gradients , a recursion relation is derived : E ( ( ∂yL ∂yli ) 2 ) = E ( ( ∂yL ∂yl+1i ) 2 ) σ2w ∫ 1√ 2π e− z2 2 φ′ ( √ q∗z ) dz ( 3 ) where φ′ the derivative of the activation function and σ2w denotes the variance of the weights . By initializing with σ2w = 2 fan−in , and considering that for the ReLU activation function φ ′ ( z ) = 1z > 0 , we have that E ( ( ∂yL ∂yli ) 2 ) = E ( ( ∂yL ∂yl+1i ) 2 ) . ( 4 ) Mean field analysis of residual networks is presented in Yang & Schoenholz ( 2017 ) , where it is shown that ResNets exhibit sub-exponential , or even polynomial dynamics , depending on the activation function . While mean field theory captures the “ average ” dynamics of random neural networks , it does not take into account the scale of gradient fluctuations that are crucial to the stability of gradient descent , which is the focus of this work . A more refined analysis presented in Pennington et al . ( 2017 ) , considers the full spectrum of the input output Jacobian of infinitely wide networks , given by : JIO = ∂yL ∂y0 ( 5 ) Exploding or vanishing gradients are then prevented by analyzing conditions that give rise to dynamical isometry , a state in which the squared eigenvalues of JIO are all concentrated around 1 . In ( Pennington et al. , 2017 ) it is shown that fully connected ReLU networks are incapable of reaching dynamical isometry , as apposed to sigmoidal networks , while in ( Pennington et al. , 2018 ) it is shown that dynamical isometry is achieved in a universal manner for a variety of activation functions . Dynamical isometry is a much stronger condition than the Jacobian norm , since it requires access to the entire spectrum . However , in previous work , this is done using various forms of infinite width approximations . Instead , this work focuses on finite depth corrections to the statistics of various quantities concerning the squared Frobenius norm of the per-layer Jacobian ( derivative of the output with respect to the weights ) , and can be viewed as a complementary approach to analyzing the dynamics at the start of training . Our results can not be obtained using large width approximations , since we seek to characterise the effect of both depth and width on the Jacobian . A recent line of work , more related to the analysis presented in this paper , deals with the statistical properties of the norm of activations and gradients in random finite width networks . Hanin & Rolnick ( 2018 ) tackle two failure modes that are caused by exponential explosion or decay of the norm of intermediate layers . It is shown that for random fully connected vanilla ReLU networks , the variance of the squared norm of the activations exponentially increases , even when initializing with the 2fan−in initialization . For ResNets , this failure mode can be overcome by correctly rescaling the residual branches . However , it is not clear how such a rescaling affects the back propagation of gradients . In our work , we prove an equivalence relation between the forward- and back-propagation of norms in most random ReLU networks . In addition , we present an analysis of DenseNets , revealing a surprising norm preservation quality that is independent of depth . Hanin ( 2018 ) explores the conditions at initialization that give rise to the exploding or vanishing gradient problem , by analyzing the individual entries of the input-output Jacobian . In the case of a single output network , the layer k to output Jacobian is given by the row vector : J ( k → L ) = ( ∂y L ∂yli , i = 1 ... nk ) = ∂yL ∂yl ( 6 ) with nk denoting the width of layer k. In our work , we take a more direct approach to analyzing the full expression for the Jacobian of random networks given by : Jkij = ∂yL ∂yk ∂yk ∂wkij = 〈J ( k → L ) , ∂y k ∂wkij 〉 . ( 7 ) Moreover , the analysis presented by Hanin ( 2018 ) does not directly carry over to ResNets or DenseNets . 3 PRELIMINARIES AND NOTATIONS . We make use of the following notations : N denotes a neural network architecture with intermediate outputs yl of layer l for a fixed input y0 . yli denotes the i ’ th component of the vector y l , and n1 ... nL denote the width of the corresponding layers , with n the length of the input vector . ‖ · ‖2 is the squared Euclidean norm , and we assume ‖y0‖2 = 1 . We denote the weight matrix associated with layer l by W l ∈ Rnl−1×nl , with lower case letters wlij denoting the individual components of W l. Additional superscripts W lk are used , when several weight matrices are associated with layer l , for instance in residual , CR and DenseNet architectures . Throughout the paper , we assume that the weights are sampled i.i.d from some symmetric distribution with moments E ( ( wlij ) m ) = clm , such that ∀ijl , ∀l , cl1 = 0 , cl2 = 2αnl−1 . We further identify the following property of a distribution ∆ = 2 ( cl4 − 3 ( cl2 ) 2 ) /nl−1 ( for Gaussian distributions , ∆ = 0 ) . The derivative of the output unit yL with respect to wlij is given by J l ij = ∂yL ∂wlij = ∂y L ∂yl ∂yl ∂wlij , with ‖J l‖2 = ∑ ij ( J l ij ) 2 . We will be concerned in our analysis with the quantities E ( ‖yl‖2 ) , var ( ‖yl‖2 ) , E ( ‖J l‖2 ) and var ( ‖J l‖2 ) , where the expectations are taken across the random sampling of the weights , given the fixed input y0 . Multi Pathway Architectures We compare a vanilla fully-connected network to ResNets He et al . ( 2016 ) and DenseNets Huang et al . ( 2016 ) to a network that is built with CR Shang et al . ( 2016 ) layers . For the standard vanilla fully-connected network , the output for of layer l is given by yl = φ ( W l > yl−1 ) , ( 8 ) where φ ( ) denotes the ReLU non-linearity . For residual architectures , we study models with m layer residual branches of the form yl = yl−1 +Rm ( yl−1 ) ( 9 ) where : Rm ( yl−1 ) = W lm > ... φ ( W l2 > φ ( W l1 > yl−1 ) . ( 10 ) We denote the k ’ th intermediate output of the l ’ th residual branch by ỹl−1 , k = W lk > ... φ ( W l2 > φ ( W l1 > yl−1 ) , ( 11 ) so that , for example Rm ( yl−1 ) = W lm > ỹl−1 , m−1 = W lm > φ ( W lm−1 > ỹl−1 , m−2 ) . ( 12 ) Furthermore , since the last layer of any residual branch W lm does not end with a non-linearity , we assume clm2 = 1 n . DenseNets were recently introduced , demonstrating faster training , as well as improved performance on several popular datasets . The main architectual features introduced by DenseNets include the connection of each layer output to all subsequent layers , using concatenation operations instead of summation , such that the weights of layer l multiply the concatenation of the outputs y0 ... yl−1 , and is of dimensionsW l ∈ Rln×n . For our analysis , we breakW l into l−1 weight matrices W l1 ... W ll−1 ∈ Rn×n , so that the output of layer l is given by yl = φ ( ∑l−1 l′=1W ll′ > yl ′ ) .

Summary: This paper studies the effects of residual and dense net type connections on the moments of per-layer gradients at random initialization. In particular, using duality, bounds on the variance of the square norm of Jacobian (with respect to the randomness of random initialization) are derived for vanilla networks. By noticing that connections that bypass a specific layer does not affect the expected square norm of the gradients of a particular layer, the paper continue to characterize the bounds for residual networks and densely connected networks that have special skip connections. In particular, with properly chosen initialization scales for each layer, the architectures with skip connections can be initialized so that the gradient norm does not explode with increasing depth.