The Singular Values of Convolutional Layers

1 Introduction . Exploding and vanishing gradients ( Hochreiter , 1991 ; Hochreiter et al. , 2001 ; Goodfellow et al. , 2016 ) are fundamental obstacles to effective training of deep neural networks . Many deep networks used in practice are layered . We can think of such networks as the composition of a number of feature transformations , followed by a linear classifier on the final layer of features . The singular values of the Jacobian of a layer bound the factor by which it increases or decreases the norm of the backpropagated signal . If these singular values are all close to 1 , then gradients neither explode nor vanish . These singular values also bound these factors in the forward direction , which affects the stability of computations , including whether the network produces the dreaded “ Nan ” . Moreover , it has been proven ( Bartlett et al. , 2017 ) that the generalization error for a network is bounded by the Lipschitz constant of the network , which in turn can be bounded by the product of the operator norms of the Jacobians of the layers . Cisse et al . ( 2017 ) discussed robustness to adversarial examples as a result of bounding the operator norm . These considerations have led authors to regularize networks by driving down the operator norms of network layers ( Drucker and Le Cun , 1992 ; Hein and Andriushchenko , 2017 ; Yoshida and Miyato , 2017 ; Miyato et al. , 2018 ) . Orthogonal initialization ( Saxe et al. , 2013 ; Pennington et al. , 2017 ) and Parseval networks ( Cisse et al. , 2017 ) are motivated by similar considerations . Convolutional layers ( LeCun et al. , 1998 ) are key components of modern deep networks . They compute linear transformations of their inputs . The Jacobian of a linear transformation is always equal to the linear transformation itself . Because of the central importance of convolutional layers to the practice of deep learning , and the fact that the singular values of the linear transformation computed by a convolutional layer are the key to its contribution to exploding and vanishing gradients , we study these singular values . Up until now , authors seeking to control the operator norm of convolutional layers have resorted to approximations ( Yoshida and Miyato , 2017 ; Miyato et al. , 2018 ; Gouk et al. , 2018a ) . In this paper , we provide an efficient way to compute the singular values exactly — this opens the door to various regularizers . We consider the convolutional layers commonly applied to image analysis tasks . The input to a typical layer is a feature map , with multiple channels for each position in an n × n field . If there are m channels , then the input as a whole is a m × n × n tensor . The output is also an n × n field with multiple channels per position1 . Each channel of the output is obtained by taking a linear combination of the values of the features in all channels in a local neighborhood centered at the corresponding position in the input feature map . Crucially , the same linear combination is used for all positions in the feature map . The coefficients are compiled in the kernel of the convolution . If the neighborhood is a k× k region , a kernelK is a k× k×m×m tensor . The projectionK : , : ,c , : gives the coefficients 1Here , to keep things simple , we are concentrating on the case that the stride is 1. that determine the cth channel of the output , in terms of the values found in all of the channels of all positions in it neighborhood ; K : , : ,c , d gives the coefficients to apply to the dth input channel , and Kp , q , c , d is the coefficient to apply to this input at in the position in the field offset horizontally by p and vertically by q . For ease of exposition , we assume that feature maps and local neighborhoods are square and that the number of channels in the output is equal to the number of channels in the input - the extension to the general case is completely straightforward . To handle edge cases in which the offsets call for inputs that are off the feature maps , practical convolutional layers either do not compute outputs ( reducing the size of the feature map ) , or pad the input with zeros . The behavior of these layers can be approximated by a layer that treats the input as if it were a torus ; when the offset calls for a pixel that is off the right end of the image , the layer “ wraps around ” to take it from the left edge , and similarly for the other edges . The quality of this approximation has been heavily analyzed in the case of one-dimensional signals ( Gray , 2006 ) . Consequently , theoretical analysis of convolutions that wrap around has been become standard . This is the case analyzed in this paper . Summary of Results : Our main result is a characterization of the singular values of a convolutional layer in terms of the kernel tensorK . Our characterization enables these singular values to be computed exactly in a simple and practically fast way , using O ( n2m2 ( m+ log n ) ) time . For comparison , the brute force solution that performs SVD on the matrix that encodes the convolutional layer ’ s linear transformation would take O ( ( n2m ) 3 ) = O ( n6m3 ) time , and is impractical for commonly used network sizes . As another point of comparison , simply to compute the convolution takes O ( n2m2k2 ) time . We prove that the following two lines of NumPy correctly compute the singular values . def SingularValues ( kernel , input_shape ) : transforms = np.fft.fft2 ( kernel , input_shape , axes= [ 0 , 1 ] ) return np.linalg.svd ( transforms , compute_uv=False ) Here kernel is any k × k ×m×m tensor2 and input_shape is the shape of the feature map to be convolved . A TensorFlow implementation is similarly simple . Timing tests , reported in Section 4.1 , confirm that this characterization speeds up the computation of singular values by multiple orders of magnitude – making it usable in practice . The algorithm first performsm2 FFTs , and then it performs n2 SVDs . The FFTs , and then the SVDs , may be executed in parallel . Our TensorFlow implementation runs a lot faster than the NumPy implementation ( see Figure 1 ) ; we think that this parallelism is the cause . We used our code to compute the singular values of the convolutional layers of the official ResNet-v2 model released with TensorFlow ( He et al. , 2016 ) . The results are described in Appendix C. Exposing the singular values of a convolutional layer opens the door to a variety of regularizers for these layers , including operator-norm regularizers . In Section 4.2 , we evaluate an algorithm that periodically projects each convolutional layer onto a operator-norm ball . Using the projections improves the test error from 6.2 % to 5.3 % on CIFAR-10 . We evaluate bounding the operator norm with and without batchnorm and we see that regularizing the operator norm helps , even in the presence of batch normalization . Moreover , operator-norm regularization and batch normalization are not redundant , and neither dominates the other . They complement each other . Related work : Prior to our work , authors have responded to the difficulty of computing the singular values of convolutional layers in various ways . Cisse et al . ( 2017 ) constrained the matrix to have orthogonal rows and scale the output of each layer by a factor of ( 2k+1 ) − 12 , for k× k kernels . Gouk et al . ( 2018a ; b ) proposed regularizing using a per-mini-batch approximation to the operator norm . They find the largest ratio between the input and output of a layer in the minibatch , and then scale down the transformation ( thereby scaling down all of the singular values , not just the largest ones ) so that the new value of this ratio obeys a constraint . Yoshida and Miyato ( 2017 ) used an approximation of the operator norm of a reshaping ofK in place of the operator norm for the linear transformation associated with K in their experiments . They reshape the given k × k ×m×m into amk2 ×m matrix , and compute its largest singular value using a power iteration method , and use this as a substitute for the operator norm . While this provides 2The same code also works if the filter height is different from the filter width , and if the number of channels in the input is different from the number of channels of the output . a useful heuristic for regularization , the largest singular value of the reshaped matrix is often quite different from the operator norm of the linear transform associated withK . Furthermore if we want to regularize using projection onto an operator-norm ball , we need the whole spectrum of the linear transformation ( see Section 3 ) . The reshaped K has only m singular values , whereas the linear transformation hasmn2 singular values of whichmn2/2 are distinct except in rare degenerate cases . It is possible to project the reshapedK onto an operator-norm ball by taking its SVD and clipping its singular values — we conducted experiments with this projection and report the results in Section 4.4 . A close relative of our main result was independently discovered by Bibi et al . ( 2019 , Lemma 2 ) . Overview of the Analysis : If the signal is 1D and there is a single input and output channel , then the linear transformation associated with a convolution is encoded by a circulant matrix , i.e. , a matrix whose rows are circular shifts of a single row ( Gray , 2006 ) . For example , for a row a = ( a1 , a2 , a3 ) , the circulant matrix circ ( a ) generated by a is ( a0 a1 a2 a2 a0 a1 a1 a2 a0 ) . In the special case of a 2D signal with a single input channel and single output channel , the linear transformation is doubly block circulant ( see ( Goodfellow et al. , 2016 ) ) . Such a matrix is made up of a circulant matrix of blocks , each of which in turn is itself circulant . Finally , when there are m input channels and m output channels , there are three levels to the hierarchy : there is am×m matrix of blocks , each of which is doubly block circulant . Our analysis extends tools from the literature built for circulant ( Horn and Johnson , 2012 ) and doubly circulant ( Chao , 1974 ) matrices to analyze the matrices with a third level in the hierarchy arising from the convolutional layers used in deep learning . One key point is that the eigenvectors of a circulant matrix are Fourier basis vectors : in the 2D , one-channel case , the matrix whose columns are the eigenvectors is F ⊗F , for the matrix F whose columns form the Fourier basis . Multiplying by this matrix is a 2D Fourier transform . In the multi-channel case , we show that the singular values can be computed by ( a ) finding the eigenvalues of each of them2 doubly circulant matrices ( of dimensions n2 × n2 ) using a 2D Fourier transform , ( b ) by forming multiple m ×m matrices , one for each eigenvalue , by picking out the i-th eigenvalue of each of the n2 × n2 blocks , for i ∈ [ 1 .. n2 ] . The union of all of the singular values of all of thosem×m matrices is the multiset of singular values of the layer . Notation : We use upper case letters for matrices , lower case for vectors . For matrix M , Mi , : represents the i-th row andM : ,j represents the j-th column ; we will also use the analogous notation for higher-order tensors . The operator norm ofM is denoted by ||M ||2 . For n ∈ N , we use [ n ] to denote the set { 0 , 1 , . . . , n− 1 } ( instead of usual { 1 , . . . , n } ) . We will index the rows and columns of matrices using elements of [ n ] , i.e . numbering from 0 . Addition of row and column indices will be done mod n unless otherwise indicated . ( Tensors will be treated analogously . ) Let σ ( · ) be the mapping from a matrix to ( the multiset of ) its singular values . 3 Let ω = exp ( 2πi/n ) , where i = √ −1 . ( Because we need a lot of indices in this paper , our use of i to define ω is the only place in the paper where we will use i to denote √ −1 . ) Let F be the n× n matrix that represents the discrete Fourier transform : Fij = ωij . We use In to denote the identity matrix of size n× n. For i ∈ [ n ] , we use ei to represent the ith basis vector in Rn . We use ⊗ to represent the Kronecker product between two matrices ( which also refers to the outer product of two vectors ) .

Summary: In this paper, the authors derive exact formulas for computing singular values of convolutional layers of deep neural networks. By appealing to fast FFT transformations, they show that computing the singular values can be done much faster than computing the full SVD of the convolution matrix. This obviates the needs to approximate the singular values. They use these results to then devise regularization schemes for DNN layers, and show that employing this regularization helps with model performance.