Constructing Orthogonal Convolutions in an Explicit Manner

1 INTRODUCTION . A layer with an orthogonal input-output Jacobian matrix is 1-Lipschitz in the 2-norm , robust to the perturbation in input . Meanwhile , it preserves the gradient norm in back-propagating the gradient , which effectively overcomes the gradient explosion and attenuation issues in training deep neural networks . In the past years , many studies have shown that exploiting orthogonality of the Jacobian matrix of layers in neural networks can achieve provable robustness to adversarial attacks ( Li et al. , 2019 ) , stabler and faster training ( Arjovsky et al. , 2016 ; Xiao et al. , 2018 ) , and improved generalization ( Cogswell et al. , 2016 ; Bansal et al. , 2018 ; Sedghi et al. , 2019 ) . In a fully-connected layer y = Wx where W ∈ Rcout×cin is the weight matrix , the layer ’ s inputoutput Jacobian matrix J = ∂y∂x is just W. Thus , preserving the orthogonality of J can be accomplished through imposing the orthogonal constraint on W , which has been extensively studied in previous works ( Mhammedi et al. , 2017 ; Cissé et al. , 2017 ; Anil et al. , 2019 ) . In contrast , in a convolution layer , the Jacobian matrix is no longer the weight matrix ( convolution kernel ) . Instead , it is the circulant matrix composed of convolution kernel ( Karami et al. , 2019 ; Sedghi et al. , 2019 ) . Thus , generally , simply constructing an orthogonal convolution kernel can not achieve an orthogonal convolution . Achieving an orthogonal Jacobian matrix in the convolution layer is more challenging than that in a fully-connected layer . It is plausibly straightforward to expand the convolution kernel to the doubly block circulant Jacobian matrix and directly impose the orthogonal constraint on the Jacobian matrix . Nevertheless , it is extremely difficult to construct a Jacobian matrix that is both doubly block-circulant and orthogonal . Block Convolutional Orthogonal parameterization ( BCOP ) ( Li et al. , 2019 ) is one of the pioneering works for constructing the orthogonal convolution neural networks . It adopts the construction algorithm ( Xiao et al. , 2018 ) which decomposes a 2-D convolution into a stack of 1-D convolutions and a channel-wise orthogonal transformation . Trockman & Kolter ( 2021 ) maps the convolution kernel and the feature tensor into the frequency domain using Fast Fourier Transform ( FFT ) , and achieves the orthogonality of the Jacobian matrix through Cayley transform on the weight matrix in the frequency domain . They devise the convolution kernel of the same size as the input feature map , which takes more parameters than standard convolution layers with a local reception field . Meanwhile , FFT maps the real-value feature map and the weight matrix into matrices of complex values , increasing the computational cost to 4 times its counterpart in the real-value domain . Meanwhile , the Cayley transform requires computing the matrix inverse , which is not friendly for GPU computation . Recently , Skew Orthogonal Convolutions ( SOC ) ( Singla & Feizi , 2021b ) devises skew-symmetric filter and exploits matrix exponential ( Hoogeboom et al. , 2020 ) to attain the orthogonality of the Jacobian matrix . But SOC is slow in evaluation since it needs to apply a convolution filter multiple times sequentially on the feature map to obtain the Taylor expansion . Observing the efficiency limitations of the existing methods , in this work , we propose an explicitly constructed orthogonal ( ECO ) convolution , which is fast in both training and evaluation . It is well known that the Jacobian matrix of a layer is orthogonal if and only if each of its singular values is 1 . Thus we convert the problem of ensuring the orthogonality of the Jacobian matrix into making every singular value as 1 . Based on the relation between the singular values of the Jacobian matrix for the convolution layer and the weight matrix of the convolution layer discovered by Sedghi et al . ( 2019 ) , we construct the convolution kernel so that it ensures every singular value of the Jacobian matrix to be 1 . Compared with the recent state-of-the-art method , SOC ( Singla & Feizi , 2021b ) implicitly approximating orthogonal convolution by multiple times convolution operations , ours explicitly builds the orthogonal convolution . Thus , we can directly deploy the constructed orthogonal convolution in evaluation , taking the same computational cost as the standard convolution . It is more efficient than SOC with multiple times convolution operations in evaluation . Experiments on CIFAR10 and CIFAR100 show that , in evaluation , taking less time , ours achieves competitive standard and robust accuracy compared with SOC . 2 RELATED WORK . Weight orthogonalization . In a fully-connected layer , the input-output Jacobian matrix is the transform itself . Thus many efforts are devoted to orthogonalizing the weights . Some early works exploit an orthogonal weight initialization to speed up the training ( Saxe et al. , 2014 ; Pennington et al. , 2017 ) and build extremely deep neural networks ( Xiao et al. , 2018 ) . Recently , more efforts are devoted to exploiting the orthogonality not only in initialization but also in the whole training process . Some approaches propose “ soft ” constraints on the weight matrix . For example , Bansal et al . ( 2018 ) ; Xiong et al . ( 2016 ) introduce mutual coherence and spectral restricted isometry as orthogonal regularization on the weight matrix . Parseval Networks ( Cissé et al. , 2017 ) adapts a regularizer to encourage the orthogonality of the weight matrices . However , these methods can not guarantee the exact orthogonality of the weight matrix . Some approaches orthogonalize features during the forward pass . For example , Huang et al . ( 2018b ) extends Batch Normalization ( Ioffe & Szegedy , 2015 ) with ZCA ; ( Huang et al. , 2018a ) solves a suboptimization problem to decorrelate deep features . These methods can not avoid computationally expensive operations like SVD , thus slow in practice . Some approaches enforce orthogonality by using the Riemannian optimization on the Stiefel manifold . For example , Casado & Martı́nez-Rubio ( 2019 ) uses Pade approximation as an alternative to matrix exponential mapping to update the weight matrix on the Stifel manifold . Li et al . ( 2020 ) uses an iterative Cayley transform to enforce weight matrix orthonormal . Anil et al . ( 2019 ) also attains orthogonal weights by orthogonal parameterization based on Björck orthonormalization ( BjoRick & Bowie , 1971 ) . Lastly , some approaches incorporate orthogonal constraint into their network architecture . For example , Mhammedi et al . ( 2017 ) builds orthgonal layers in RNNs with householder reflections . Orthogonal convolution . In a convolution layer , the Jacobian matrix is no longer the weight matrix . Thus , the above-mentioned weight orthogonalization methods can not be trivially applied in convolution layers . Wang et al . ( 2020 ) ; Qi et al . ( 2020 ) encourage the orthogonality of the Jacobian matrix of convolution layers through a regularizer , but they can not achieve the strict orthogonality and can not attain the provable robustness . Block Convolutional Orthogonal parameterization ( BCOP ) ( Li et al. , 2019 ) is a pioneering work for enforcing the orthogonality of the Jacobian matrix of a convolution layer . It conducts parameterization of orthogonal convolutions by adapting the construction algorithm proposed by Xiao et al . ( 2018 ) . But BCOP is slow in training . Trockman & Kolter ( 2021 ) applies the Cayley transform to a skew-symmetric convolution weight in the Fourier domain so that the convolution recovered from the Fourier domain has an orthogonal Jacobian matrix . Though it achieves higher efficiency in some applications than BCOP , it is still significantly more costly than standard convolutional layers . Skew Orthogonal Convolution ( Singla & Feizi , 2021b ) achieves orthogonal convolution through Taylor expansion of the matrix exponential . It is faster than BCOP in training but slower in evaluation . In contrast , our method is efficient in both training and evaluation . Recently , Su et al . ( 2021 ) achieves orthogonal convolutions via paraunitary systems in an elegant manner . Nevertheless , it also suffers from inefficiency issue . 3 PRELIMINARY . Notation . For a matrix M , M [ i , j ] denotes the element in the i-th row and the j-th column , M [ i , : ] denotes the i-th row and M [ : , j ] denotes the j-th column . We use the similar notation for indexing tensors . For n ∈ N , [ n ] denotes the set of n numbers from 0 to n − 1 , that is , { 0 , 1 , · · · , n − 1 } . We denote the function to obtain the singular values by σ ( · ) . Let vec ( · ) denotes the function which unfolds a matrix or higher-order tensor into a vector . Let ωk = exp ( 2πi/k ) where i = √ −1 . Let Fk ∈ Rk×k be matrix for the discrete Fourier transform ( DFT ) for k-element vectors and each entry of Fk is defined as Fk [ p , q ] = ω pq k . 3.1 RELATION BETWEEN THE JACOBIAN MATRIX AND THE WEIGHT MATRIX . We denote the input tensor of a convolution layer by X ∈ Rc×n×n , where n is the spatial size , and c is the number of input channels . We denote the convolution operation on X by convŴ ( X ) , where Ŵ ∈ Rk×k×c×c size where k is the size of the reception field and k is normally smaller than size of the input image , n. In this definition , we set the number of output channels identical to that of input channels . Meanwhile , we set the convolution stride as 1 and adapt the cyclic padding to make the output of the same size as the input . We denote the output tensor by Y ∈ Rc×n×n and each element of Y is obtained by ∀l , s , t ∈ [ c ] × [ n ] × [ n ] , Y [ l , s , t ] = ∑ r∈ [ c ] ∑ p∈ [ k ] ∑ q∈ [ k ] X [ l + r , s+ p , t+ q ] Ŵ [ p , q , l , r ] . ( 1 ) Since the convolution operation is a linear transformation , there exists a linear transformation M ∈ Rn2c×n2c which satisfies : Y = convŴ ( X ) ⇔ vec ( Y ) = Mvec ( X ) . ( 2 ) To simplify illustration , Ŵ ∈ Rk×k×c×c is normally expanded into W ∈ Rn×n×c×c through zeropadding . We term W as the expanded convolution kernel . In W , only k2c2 elements are non-zero . Based on the above definition , we review the following theorem in Sedghi et al . ( 2019 ) : Theorem 1 ( see Sedghi et al . ( 2019 ) Section 2.2 ) . For any expanded convolution kernel W ∈ Rn×n×c×c , let M is the matrix encoding the linear transformation of a convolution with W . For each p , q ∈ [ n ] × [ n ] , let P ( p , q ) be the c× c matrix computed by ∀s , t ∈ [ c ] × [ c ] , P ( p , q ) [ s , t ] = ( F > nW [ : , : , s , t ] Fn ) [ p , q ] . ( 3 ) Then σ ( M ) = ⋃ p∈ [ n ] , q∈ [ n ] σ ( P ( p , q ) ) . ( 4 ) The above theorem gives the relation between the singular values of the transformation matrix M and the expanded convolution W . Theorem 2 . For any W ∈ Rn×n×c×c , the Jacobian matrix of the convolution layer , J , is orthogonal if and only if ∀p , q ∈ [ n ] × [ n ] , each singular value of the matrix P ( p , q ) is 1 . Proof . A real matrix is orthogonal if and only if each singular value is 1 . Meanwhile , the Jacobian matrix J = ∂vec ( Y ) ∂vec ( X ) = M. Based on Theorem 1 , each singular value of M is 1 if and only if ∀p , q ∈ [ n ] × [ n ] , each singular value of the matrix P ( p , q ) is 1 , thus we complete the proof .

Summary: This paper studies how to construct orthogonal convolutional networks in an efficient way. To this end, this paper builds the connection between the DFT-transformed kernel with the common dilated convolution. During training, the forward pass can be done by a sequence of inverse DFT and dilated convolution. During testing, all the convolution kernels only need to be transformed once so that evaluation time is significantly reduced.