A Johnson-Lindenstrauss Framework for Randomly Initialized CNNs

1 INTRODUCTION . Neural networks have become a standard tool in multiple scientific fields , due to their success in classification ( and estimation ) tasks . Conceptually , this success is achieved since better representation is allowed by each subsequent layer until linear separability is achieved at the last ( linear ) layer . Indeed , in many disciplines involving real-world tasks , such as computer vision and natural language processing , the training process is biased toward these favorable representations . This bias is a product of several factors , with the neural-network initialization playing a pivotal role ( Sutskever et al. , 2013 ) . Therefore , we concentrate in this work on studying the initialization of neural networks , with the following question guiding this work . How does the geometric representation of a dataset change after the application of each randomly initialized layer of a neural network ? To answer this , we study how the following two geometrical quantities change after each layer . 〈x , y〉 The scalar inner product between vectors x and y.1 ρ : = 〈x , y〉‖x‖‖y‖ The cosine similarity ( or simply similarity ) between vectors x and y . The similarity ρ ∈ [ −1 , 1 ] between x and y equals ρ = cos ( θ ) , where θ is the angle between them . Consider first one layer of a fully connected neural network ( FNN ) with an identity activation ( linear FNN ) that is initialized by independent identically distributed ( i.i.d . ) Gaussian weights with mean zero and variance 1/N , where N is the number of neurons in that layer . This random linear FNN induces an isometric embedding of the dataset , namely , the similarity ρin between any two inputs , xin and yin , is preserved together with their norm : ρout ≈ ρ̄out = ρin , where ρout is the similarity between the resulting random ( due to the multiplication by the random weights ) outputs , Xout and Yout ( respectively ) , and ρ̄out is the mean output similarity defined by 1We mean here a vector in a wider sense : x and y may be matrices and tensors ( of the same dimensions ) . In this situation , the standard inner product is equal to the vectorization thereof : 〈x , y〉 = 〈vec ( x ) , vec ( y ) 〉 . ρout : = 〈Xout , Yout〉 ‖Xout‖ ‖Yout‖ , and ρ̄out : = E [ 〈Xout , Yout〉 ] √ E [ ‖Xout‖2 ] E [ ‖Yout‖2 ] . ( 1 ) The proof of this isometric relation between the input and output similarities follows from the celebrated Johnson–Lindenstrauss lemma ( Johnson and Lindenstrauss , 1984 ; Dasgupta and Gupta , 1999 ) . This lemma states that a random linear map of dimension N preserves the distance between any two points up to an ǫ > 0 contraction/expansion with probability at least 1 − δ for all N > c log ( 1/δ ) /ǫ2 for an absolute constant c. In the context of randomly initialized linear FNNs , this result means that , for a number of neurons N that satisfies N > c log ( 1/δ ) /ǫ2 , P ( |ρout − ρin| < ǫ ) ≥ 1− δ So , conceptually , the Johnson–Lindenstrauss lemma studies how inner products ( or geometry ) change , in expectation , after applying a random transformation and how well an average of these random transformations is concentrated around the expectation . This is the exact setting of randomly initialized neural networks . The random transformations consist of a random projection ( multiplying the dataset by a random matrix ) which is followed by a non-linearity . Naturally , adding a non-linearity complicates the picture . Let us focus on the case where the activation function is a rectified linear unit ( ReLU ) . That is , consider a random fully-connected layer with ReLU activation initialized with i.i.d . zero-mean Gaussian weights and two different inputs . For this case , Cho and Saul ( 2009 ) , Giryes et al . ( 2016 ) , and Daniely et al . ( 2016 ) proved that2 ρout ≈ ρ̄out = √ 1− ρ2in + ( π − cos−1 ( ρin ) ) ρin π . ( 2 ) Following Daniely et al . ( 2016 ) , we refer to the resulting function in ( 2 ) of ρ̄out in ρin as the dual activation of the ReLU ; this function is represented in figure 1 by the yellow curve . One may easily verify that the dual activation ( 2 ) of the ReLU satisfies ρout > ρin for ρin 6= 1 , meaning that it is a contraction . Consequently , for deep FNNs , which comprise multiple layers , random initialization results in a collapse of all inputs with the same norm ( a sphere ) to a single point at the output of the FNN ( equivalently , the entire dataset collapses to a single straight line ) . Intuitively , this collapse is an unfavorable starting point for optimization . To see why , consider the gradient ∇ w ( i ) j of the weight w ( i ) j of some neuron j in a deep layer i in a randomly initialized ReLU FNN . By the chain rule ( backpropagation ) , this gradient is proportional to the output of the previous layer a ( i−1 ) for the corresponding input , i.e. , it holds that ∇ w ( i ) j ∝ a ( i−1 ) . If the collapse is present already at layer i , this output is essentially proportional to a fixed vector a∗ . But this implies that , in the gradient update , the weights of the deep layer will move roughly along a straight line which would impede , in turn , the process of achieving linear separability . Indeed , it is considered that for a FNN to train well , its input–output Jacobian needs to exhibit dynamical isometry upon initialization ( Saxe et al. , 2014 ) . Namely , the singular values of the Jacobian ∂xout/∂xin must be concentrated around 1 , where xout and xin denote the input and output of the FNN , respectively . If the dataset collapses to a line , xout is essentially invariant to xin ( up to a change in its norm ) , suggesting that the singular values of ∂xout/∂xin are close to zero . Therefore , randomly initialized FNNs exhibit the opposite behavior from dynamical isometry and hence do not train well . 1.1 OUR CONTRIBUTION . Our main interest lies in the following question . Does the contraction observed in randomly initialized ReLU FNNs carry over to convolutional neural networks ( CNNs ) ? As we will show , qualitatively , the answer is yes . However , quantitatively , the answer is more subtle as is illustrated in figure 1 . In this figure , the similarity between pairs of inputs sampled at random 2The results in ( Cho and Saul , 2009 ) and ( Daniely et al. , 2016 ) were derived assuming unit-norm vectors ‖xin‖ = ‖yin‖ = 1 . The result here follows by the homogeneity of the ReLU activation function : R ( αx ) = αR ( x ) for α ≥ 0 , and ergodicity , assuming multiple filters are applied . from standard natural image datasets—Fashion MNIST ( F-MNIST ) , CIFAR-10 , and ImageNet—are displayed against the corresponding output of a randomly initialized CNN layer . For these datasets , clearly ρout ≈ ρin meaning that the relation of ReLU FNNs ( 2 ) —represented in figure 1 by the yellow curve—breaks down . That said , for inputs consisting of i.i.d . zero-mean Gaussians ( and filters comprising i.i.d . zero-mean Gaussian weights as before ) with a Pearson correlation coefficient ρ between corresponding entries ( and independent otherwise ) , the relation in ( 2 ) between ρ̄out and ρin of ReLU FNNs does hold for ReLU CNNs as well , as illustrated in figure 1a . This dataset-dependent behavior , observed in figure 1 , suggests that , in contrast to randomlyinitialized FNNs which behave according to ( 2 ) , randomly-initialized CNNs exhibit a richer behavior : ρ̄out does not depend only on ρin but on the inputs xin and yin themselves . Therefore , in this work , we characterize the behavior of ρ̄out after applying one layer in randomly initialized CNNs . We start by considering randomly initialized CNNs with general activation functions . We show in theorem 1 that the expected ( over the filters ) inner product E [ 〈Xout , Yout〉 ] and the mean similarity ρ̄out depend on xin and yin ( and not just 〈xin , yin〉 ) by extending the dual-activation notion of Daniely et al . ( 2016 ) . In theorem 2 , we further prove that , by taking multiple independent filters , 〈Xout , Yout〉 and ρout of ( 1 ) concentrate around E [ 〈Xout , Yout〉 ] and ρ̄out , respectively . We then specialize these results to linear CNNs ( with identity activation ) and derive a convolutionbased variant of the Johnson–Lindenstrauss lemma that shows that 〈Xout , Yout〉 ≈ 〈xin , yin〉 and ρout ≈ ρin for linear CNNs , both in expectation ( ρ̄out = ρin for the latter ) and with high probability . For randomly initialized ReLU CNNs , we derive the following tight upper and lower bounds for ρ̄out in terms of ρin in theorem 3 : max { ρin , 0 } ≤ ρ̄out ≤ 1 + ρin 2 . ( 3 ) These bounds imply , in turn , that for ρin 6= 1 each ReLU CNN layer is contracting . In theorem 4 we prove that ρ̄out for random Gaussian data satisfies the relation ( 2 ) , in accordance with figure 1a . To explain the ( almost ) isometric behavior of CNNs for natural images ( figure 1 ) , we note that many natural images consist of large , relative to the filter size , approximately monochromatic patches . This observation leads to a simple model of black and white ( binary ) images with “ large patches ” . To describe this model mathematically , we define a notion of a shared boundary between two images in definition 2 , and model large patches by bodies whose area is large compared to the shared boundary . We prove that ρ̄out ≈ ρin for this model , meaning that the lower bound in ( 3 ) is in fact tight . 1.2 RELATED WORK . In this paper , we study how various inputs are embedded by randomly initialized convolutional neural networks . Neural tangent kernel ( NTK ) ( Jacot et al. , 2018 ) is a related line of work . This setting studies the infinite width limit ( among other assumptions ) in which one can consider neural network training as regression over a fixed kernel ; this kernel is the NTK . There are two factors that affect the calculation of the NTK : The embedding of the input space at initialization and the gradients at initialization . In this paper we study the first one . Arora et al . ( 2019 ) , and Bietti and Mairal ( 2019 ) give expressions for the NTK and the convolutional NTK ( CNTK ) . Theorem 1 may be implicitly deduced from those expressions . Arora et al . ( 2019 ) provide concentration bounds for the NTK of fully connected networks with finite width . Bietti and Mairal ( 2019 ) derive smoothness properties for NTKs , e.g. , upper bounds on the deformation induced by the NTK in terms of the initial Euclidean distance between the inputs . A related approach to NTK is taken in ( Bietti , 2021 ) where convolutional kernel networks ( CKN ) are used . Standard initialization techniques use the Glorot initializtion ( Glorot and Bengio , 2010 ) and He initialization ( He et al. , 2015 ) . Both were introduced to prevent the gradients from exploding/vanishing . On a similar note , Hanin and Rolnick ( 2018 ) discuss how to prevent exploding/vanishing mean activation length—which corresponds to the gradients to some degree—in FNN with and without skip connections . For a comprehensive review on other techniques see ( Narkhede et al. , 2021-06-28 ) . Schoenholz et al . ( 2017 ) ; Poole et al . ( 2016 ) ; Yang and Schoenholz ( 2017 ) study the initialization of FNNs using mean-field theory , in a setting where the width of the network is infinite . They demonstrate that for some activation functions there exists an initialization variance such that the network does not suffer from vanishing/exploding gradients . In this case , the network is said to be initialized at the edge of chaos . As mentioned earlier , Saxe et al . ( 2014 ) introduced a stronger requirement than that of moderate gradients at initialization , in the form of dynamical isometry . for linear FNNs , they showed that orthogonal initialization achieves dynamical isometry whereas Gaussian i.i.d . initialization does not . For non-linear FNNs , Pennington et al . ( 2017 ) show that the hyperbolic tangent ( tanh ) activation can achieve dynamical isometry while ReLU FNNs with Gaussian i.i.d . initialization or orthogonal initialization can not . In contrast , Burkholz and Dubatovka ( 2019 ) show that ReLU FNNs achieve dynamical isometry by moving away from i.i.d . initialization . And lastly , Tarnowski et al . ( 2019 ) show that residual networks achieve dynamical isometry over a broad spectrum of activations ( including ReLU ) and initializations . Xiao et al . ( 2018 ) trained tanh ( and not ReLU ) CNNs with 10000 layers that achieve dynamical isometry using the delta-orthogonal initialization . Similarly , Zhang et al . ( 2019 ) trained residual CNNs with 10000 layers without batch normalization using fixup-initialization that prevents exploding/vanishing gradients .

Summary: The paper shows (as clearly described in the introduction) two things: random filters have the distance preservation property. For random convolution + ReLU activation it is contracted (which is also expected, since ReLU throws away half of the entries). However, for images the authors try to explain why CNN+ReLU is isometric by considering images as objects with large monochromatic images.   The main motivation for such research in practice is initialization. A deep fully-connected network with ReLU may lead to collapse, whereas for CNNs it does not. The paper argues that the randomized theory presented here is a justification for such fact for images.