A Johnson-Lindenstrauss Framework for Randomly Initialized CNNs

1 INTRODUCTION . Neural networks have become a standard tool in multiple scientific fields , due to their success in classification ( and estimation ) tasks . Conceptually , this success is achieved since better representation is allowed by each subsequent layer until linear separability is achieved at the last ( linear ) layer . Indeed , in many disciplines involving real-world tasks , such as computer vision and natural language processing , the training process is biased toward these favorable representations . This bias is a product of several factors , with the neural-network initialization playing a pivotal role ( Sutskever et al. , 2013 ) . Therefore , we concentrate in this work on studying the initialization of neural networks , with the following question guiding this work . How does the geometric representation of a dataset change after the application of each randomly initialized layer of a neural network ? To answer this , we study how the following two geometrical quantities change after each layer . 〈x , y〉 The scalar inner product between vectors x and y.1 ρ : = 〈x , y〉‖x‖‖y‖ The cosine similarity ( or simply similarity ) between vectors x and y . The similarity ρ ∈ [ −1 , 1 ] between x and y equals ρ = cos ( θ ) , where θ is the angle between them . Consider first one layer of a fully connected neural network ( FNN ) with an identity activation ( linear FNN ) that is initialized by independent identically distributed ( i.i.d . ) Gaussian weights with mean zero and variance 1/N , where N is the number of neurons in that layer . This random linear FNN induces an isometric embedding of the dataset , namely , the similarity ρin between any two inputs , xin and yin , is preserved together with their norm : ρout ≈ ρ̄out = ρin , where ρout is the similarity between the resulting random ( due to the multiplication by the random weights ) outputs , Xout and Yout ( respectively ) , and ρ̄out is the mean output similarity defined by 1We mean here a vector in a wider sense : x and y may be matrices and tensors ( of the same dimensions ) . In this situation , the standard inner product is equal to the vectorization thereof : 〈x , y〉 = 〈vec ( x ) , vec ( y ) 〉 . ρout : = 〈Xout , Yout〉 ‖Xout‖ ‖Yout‖ , and ρ̄out : = E [ 〈Xout , Yout〉 ] √ E [ ‖Xout‖2 ] E [ ‖Yout‖2 ] . ( 1 ) The proof of this isometric relation between the input and output similarities follows from the celebrated Johnson–Lindenstrauss lemma ( Johnson and Lindenstrauss , 1984 ; Dasgupta and Gupta , 1999 ) . This lemma states that a random linear map of dimension N preserves the distance between any two points up to an ǫ > 0 contraction/expansion with probability at least 1 − δ for all N > c log ( 1/δ ) /ǫ2 for an absolute constant c. In the context of randomly initialized linear FNNs , this result means that , for a number of neurons N that satisfies N > c log ( 1/δ ) /ǫ2 , P ( |ρout − ρin| < ǫ ) ≥ 1− δ So , conceptually , the Johnson–Lindenstrauss lemma studies how inner products ( or geometry ) change , in expectation , after applying a random transformation and how well an average of these random transformations is concentrated around the expectation . This is the exact setting of randomly initialized neural networks . The random transformations consist of a random projection ( multiplying the dataset by a random matrix ) which is followed by a non-linearity . Naturally , adding a non-linearity complicates the picture . Let us focus on the case where the activation function is a rectified linear unit ( ReLU ) . That is , consider a random fully-connected layer with ReLU activation initialized with i.i.d . zero-mean Gaussian weights and two different inputs . For this case , Cho and Saul ( 2009 ) , Giryes et al . ( 2016 ) , and Daniely et al . ( 2016 ) proved that2 ρout ≈ ρ̄out = √ 1− ρ2in + ( π − cos−1 ( ρin ) ) ρin π . ( 2 ) Following Daniely et al . ( 2016 ) , we refer to the resulting function in ( 2 ) of ρ̄out in ρin as the dual activation of the ReLU ; this function is represented in figure 1 by the yellow curve . One may easily verify that the dual activation ( 2 ) of the ReLU satisfies ρout > ρin for ρin 6= 1 , meaning that it is a contraction . Consequently , for deep FNNs , which comprise multiple layers , random initialization results in a collapse of all inputs with the same norm ( a sphere ) to a single point at the output of the FNN ( equivalently , the entire dataset collapses to a single straight line ) . Intuitively , this collapse is an unfavorable starting point for optimization . To see why , consider the gradient ∇ w ( i ) j of the weight w ( i ) j of some neuron j in a deep layer i in a randomly initialized ReLU FNN . By the chain rule ( backpropagation ) , this gradient is proportional to the output of the previous layer a ( i−1 ) for the corresponding input , i.e. , it holds that ∇ w ( i ) j ∝ a ( i−1 ) . If the collapse is present already at layer i , this output is essentially proportional to a fixed vector a∗ . But this implies that , in the gradient update , the weights of the deep layer will move roughly along a straight line which would impede , in turn , the process of achieving linear separability . Indeed , it is considered that for a FNN to train well , its input–output Jacobian needs to exhibit dynamical isometry upon initialization ( Saxe et al. , 2014 ) . Namely , the singular values of the Jacobian ∂xout/∂xin must be concentrated around 1 , where xout and xin denote the input and output of the FNN , respectively . If the dataset collapses to a line , xout is essentially invariant to xin ( up to a change in its norm ) , suggesting that the singular values of ∂xout/∂xin are close to zero . Therefore , randomly initialized FNNs exhibit the opposite behavior from dynamical isometry and hence do not train well . 1.1 OUR CONTRIBUTION . Our main interest lies in the following question . Does the contraction observed in randomly initialized ReLU FNNs carry over to convolutional neural networks ( CNNs ) ? As we will show , qualitatively , the answer is yes . However , quantitatively , the answer is more subtle as is illustrated in figure 1 . In this figure , the similarity between pairs of inputs sampled at random 2The results in ( Cho and Saul , 2009 ) and ( Daniely et al. , 2016 ) were derived assuming unit-norm vectors ‖xin‖ = ‖yin‖ = 1 . The result here follows by the homogeneity of the ReLU activation function : R ( αx ) = αR ( x ) for α ≥ 0 , and ergodicity , assuming multiple filters are applied . from standard natural image datasets—Fashion MNIST ( F-MNIST ) , CIFAR-10 , and ImageNet—are displayed against the corresponding output of a randomly initialized CNN layer . For these datasets , clearly ρout ≈ ρin meaning that the relation of ReLU FNNs ( 2 ) —represented in figure 1 by the yellow curve—breaks down . That said , for inputs consisting of i.i.d . zero-mean Gaussians ( and filters comprising i.i.d . zero-mean Gaussian weights as before ) with a Pearson correlation coefficient ρ between corresponding entries ( and independent otherwise ) , the relation in ( 2 ) between ρ̄out and ρin of ReLU FNNs does hold for ReLU CNNs as well , as illustrated in figure 1a . This dataset-dependent behavior , observed in figure 1 , suggests that , in contrast to randomlyinitialized FNNs which behave according to ( 2 ) , randomly-initialized CNNs exhibit a richer behavior : ρ̄out does not depend only on ρin but on the inputs xin and yin themselves . Therefore , in this work , we characterize the behavior of ρ̄out after applying one layer in randomly initialized CNNs . We start by considering randomly initialized CNNs with general activation functions . We show in theorem 1 that the expected ( over the filters ) inner product E [ 〈Xout , Yout〉 ] and the mean similarity ρ̄out depend on xin and yin ( and not just 〈xin , yin〉 ) by extending the dual-activation notion of Daniely et al . ( 2016 ) . In theorem 2 , we further prove that , by taking multiple independent filters , 〈Xout , Yout〉 and ρout of ( 1 ) concentrate around E [ 〈Xout , Yout〉 ] and ρ̄out , respectively . We then specialize these results to linear CNNs ( with identity activation ) and derive a convolutionbased variant of the Johnson–Lindenstrauss lemma that shows that 〈Xout , Yout〉 ≈ 〈xin , yin〉 and ρout ≈ ρin for linear CNNs , both in expectation ( ρ̄out = ρin for the latter ) and with high probability . For randomly initialized ReLU CNNs , we derive the following tight upper and lower bounds for ρ̄out in terms of ρin in theorem 3 : max { ρin , 0 } ≤ ρ̄out ≤ 1 + ρin 2 . ( 3 ) These bounds imply , in turn , that for ρin 6= 1 each ReLU CNN layer is contracting . In theorem 4 we prove that ρ̄out for random Gaussian data satisfies the relation ( 2 ) , in accordance with figure 1a . To explain the ( almost ) isometric behavior of CNNs for natural images ( figure 1 ) , we note that many natural images consist of large , relative to the filter size , approximately monochromatic patches . This observation leads to a simple model of black and white ( binary ) images with “ large patches ” . To describe this model mathematically , we define a notion of a shared boundary between two images in definition 2 , and model large patches by bodies whose area is large compared to the shared boundary . We prove that ρ̄out ≈ ρin for this model , meaning that the lower bound in ( 3 ) is in fact tight . 1.2 RELATED WORK . In this paper , we study how various inputs are embedded by randomly initialized convolutional neural networks . Neural tangent kernel ( NTK ) ( Jacot et al. , 2018 ) is a related line of work . This setting studies the infinite width limit ( among other assumptions ) in which one can consider neural network training as regression over a fixed kernel ; this kernel is the NTK . There are two factors that affect the calculation of the NTK : The embedding of the input space at initialization and the gradients at initialization . In this paper we study the first one . Arora et al . ( 2019 ) , and Bietti and Mairal ( 2019 ) give expressions for the NTK and the convolutional NTK ( CNTK ) . Theorem 1 may be implicitly deduced from those expressions . Arora et al . ( 2019 ) provide concentration bounds for the NTK of fully connected networks with finite width . Bietti and Mairal ( 2019 ) derive smoothness properties for NTKs , e.g. , upper bounds on the deformation induced by the NTK in terms of the initial Euclidean distance between the inputs . A related approach to NTK is taken in ( Bietti , 2021 ) where convolutional kernel networks ( CKN ) are used . Standard initialization techniques use the Glorot initializtion ( Glorot and Bengio , 2010 ) and He initialization ( He et al. , 2015 ) . Both were introduced to prevent the gradients from exploding/vanishing . On a similar note , Hanin and Rolnick ( 2018 ) discuss how to prevent exploding/vanishing mean activation length—which corresponds to the gradients to some degree—in FNN with and without skip connections . For a comprehensive review on other techniques see ( Narkhede et al. , 2021-06-28 ) . Schoenholz et al . ( 2017 ) ; Poole et al . ( 2016 ) ; Yang and Schoenholz ( 2017 ) study the initialization of FNNs using mean-field theory , in a setting where the width of the network is infinite . They demonstrate that for some activation functions there exists an initialization variance such that the network does not suffer from vanishing/exploding gradients . In this case , the network is said to be initialized at the edge of chaos . As mentioned earlier , Saxe et al . ( 2014 ) introduced a stronger requirement than that of moderate gradients at initialization , in the form of dynamical isometry . for linear FNNs , they showed that orthogonal initialization achieves dynamical isometry whereas Gaussian i.i.d . initialization does not . For non-linear FNNs , Pennington et al . ( 2017 ) show that the hyperbolic tangent ( tanh ) activation can achieve dynamical isometry while ReLU FNNs with Gaussian i.i.d . initialization or orthogonal initialization can not . In contrast , Burkholz and Dubatovka ( 2019 ) show that ReLU FNNs achieve dynamical isometry by moving away from i.i.d . initialization . And lastly , Tarnowski et al . ( 2019 ) show that residual networks achieve dynamical isometry over a broad spectrum of activations ( including ReLU ) and initializations . Xiao et al . ( 2018 ) trained tanh ( and not ReLU ) CNNs with 10000 layers that achieve dynamical isometry using the delta-orthogonal initialization . Similarly , Zhang et al . ( 2019 ) trained residual CNNs with 10000 layers without batch normalization using fixup-initialization that prevents exploding/vanishing gradients .

Summary:  The authors consider angle propagation in randomly-initialized convolutional neural network layers -- given two input images that have cosine angle $\rho$, what is the cosine angle of their feature embeddings when propagated through the random layer? They show that the behavior is very different from the standard feedforward case that has been discussed at length following the work of Daniely et al. (2016): for filters of small spatial support and e.g. ReLU activations, the output cosine angle can be as small as the ReLU of the input cosine angle, or as large as a linear function of the input cosine angle, depending on the patch structure of the input image. This has implications for understanding signal propagation and issues like dynamical isometry in convolutional networks. The authors give some measure concentration results, in the spirit of Daniely et al.'s results for feedforward networks, that specify the behavior in the convolutional case, and present examples of different types of images (unstructured/gaussian; cartoons; CIFAR-10 data) where different angle propagation behaviors are observed.