Constructing Orthogonal Convolutions in an Explicit Manner

1 INTRODUCTION . A layer with an orthogonal input-output Jacobian matrix is 1-Lipschitz in the 2-norm , robust to the perturbation in input . Meanwhile , it preserves the gradient norm in back-propagating the gradient , which effectively overcomes the gradient explosion and attenuation issues in training deep neural networks . In the past years , many studies have shown that exploiting orthogonality of the Jacobian matrix of layers in neural networks can achieve provable robustness to adversarial attacks ( Li et al. , 2019 ) , stabler and faster training ( Arjovsky et al. , 2016 ; Xiao et al. , 2018 ) , and improved generalization ( Cogswell et al. , 2016 ; Bansal et al. , 2018 ; Sedghi et al. , 2019 ) . In a fully-connected layer y = Wx where W ∈ Rcout×cin is the weight matrix , the layer ’ s inputoutput Jacobian matrix J = ∂y∂x is just W. Thus , preserving the orthogonality of J can be accomplished through imposing the orthogonal constraint on W , which has been extensively studied in previous works ( Mhammedi et al. , 2017 ; Cissé et al. , 2017 ; Anil et al. , 2019 ) . In contrast , in a convolution layer , the Jacobian matrix is no longer the weight matrix ( convolution kernel ) . Instead , it is the circulant matrix composed of convolution kernel ( Karami et al. , 2019 ; Sedghi et al. , 2019 ) . Thus , generally , simply constructing an orthogonal convolution kernel can not achieve an orthogonal convolution . Achieving an orthogonal Jacobian matrix in the convolution layer is more challenging than that in a fully-connected layer . It is plausibly straightforward to expand the convolution kernel to the doubly block circulant Jacobian matrix and directly impose the orthogonal constraint on the Jacobian matrix . Nevertheless , it is extremely difficult to construct a Jacobian matrix that is both doubly block-circulant and orthogonal . Block Convolutional Orthogonal parameterization ( BCOP ) ( Li et al. , 2019 ) is one of the pioneering works for constructing the orthogonal convolution neural networks . It adopts the construction algorithm ( Xiao et al. , 2018 ) which decomposes a 2-D convolution into a stack of 1-D convolutions and a channel-wise orthogonal transformation . Trockman & Kolter ( 2021 ) maps the convolution kernel and the feature tensor into the frequency domain using Fast Fourier Transform ( FFT ) , and achieves the orthogonality of the Jacobian matrix through Cayley transform on the weight matrix in the frequency domain . They devise the convolution kernel of the same size as the input feature map , which takes more parameters than standard convolution layers with a local reception field . Meanwhile , FFT maps the real-value feature map and the weight matrix into matrices of complex values , increasing the computational cost to 4 times its counterpart in the real-value domain . Meanwhile , the Cayley transform requires computing the matrix inverse , which is not friendly for GPU computation . Recently , Skew Orthogonal Convolutions ( SOC ) ( Singla & Feizi , 2021b ) devises skew-symmetric filter and exploits matrix exponential ( Hoogeboom et al. , 2020 ) to attain the orthogonality of the Jacobian matrix . But SOC is slow in evaluation since it needs to apply a convolution filter multiple times sequentially on the feature map to obtain the Taylor expansion . Observing the efficiency limitations of the existing methods , in this work , we propose an explicitly constructed orthogonal ( ECO ) convolution , which is fast in both training and evaluation . It is well known that the Jacobian matrix of a layer is orthogonal if and only if each of its singular values is 1 . Thus we convert the problem of ensuring the orthogonality of the Jacobian matrix into making every singular value as 1 . Based on the relation between the singular values of the Jacobian matrix for the convolution layer and the weight matrix of the convolution layer discovered by Sedghi et al . ( 2019 ) , we construct the convolution kernel so that it ensures every singular value of the Jacobian matrix to be 1 . Compared with the recent state-of-the-art method , SOC ( Singla & Feizi , 2021b ) implicitly approximating orthogonal convolution by multiple times convolution operations , ours explicitly builds the orthogonal convolution . Thus , we can directly deploy the constructed orthogonal convolution in evaluation , taking the same computational cost as the standard convolution . It is more efficient than SOC with multiple times convolution operations in evaluation . Experiments on CIFAR10 and CIFAR100 show that , in evaluation , taking less time , ours achieves competitive standard and robust accuracy compared with SOC . 2 RELATED WORK . Weight orthogonalization . In a fully-connected layer , the input-output Jacobian matrix is the transform itself . Thus many efforts are devoted to orthogonalizing the weights . Some early works exploit an orthogonal weight initialization to speed up the training ( Saxe et al. , 2014 ; Pennington et al. , 2017 ) and build extremely deep neural networks ( Xiao et al. , 2018 ) . Recently , more efforts are devoted to exploiting the orthogonality not only in initialization but also in the whole training process . Some approaches propose “ soft ” constraints on the weight matrix . For example , Bansal et al . ( 2018 ) ; Xiong et al . ( 2016 ) introduce mutual coherence and spectral restricted isometry as orthogonal regularization on the weight matrix . Parseval Networks ( Cissé et al. , 2017 ) adapts a regularizer to encourage the orthogonality of the weight matrices . However , these methods can not guarantee the exact orthogonality of the weight matrix . Some approaches orthogonalize features during the forward pass . For example , Huang et al . ( 2018b ) extends Batch Normalization ( Ioffe & Szegedy , 2015 ) with ZCA ; ( Huang et al. , 2018a ) solves a suboptimization problem to decorrelate deep features . These methods can not avoid computationally expensive operations like SVD , thus slow in practice . Some approaches enforce orthogonality by using the Riemannian optimization on the Stiefel manifold . For example , Casado & Martı́nez-Rubio ( 2019 ) uses Pade approximation as an alternative to matrix exponential mapping to update the weight matrix on the Stifel manifold . Li et al . ( 2020 ) uses an iterative Cayley transform to enforce weight matrix orthonormal . Anil et al . ( 2019 ) also attains orthogonal weights by orthogonal parameterization based on Björck orthonormalization ( BjoRick & Bowie , 1971 ) . Lastly , some approaches incorporate orthogonal constraint into their network architecture . For example , Mhammedi et al . ( 2017 ) builds orthgonal layers in RNNs with householder reflections . Orthogonal convolution . In a convolution layer , the Jacobian matrix is no longer the weight matrix . Thus , the above-mentioned weight orthogonalization methods can not be trivially applied in convolution layers . Wang et al . ( 2020 ) ; Qi et al . ( 2020 ) encourage the orthogonality of the Jacobian matrix of convolution layers through a regularizer , but they can not achieve the strict orthogonality and can not attain the provable robustness . Block Convolutional Orthogonal parameterization ( BCOP ) ( Li et al. , 2019 ) is a pioneering work for enforcing the orthogonality of the Jacobian matrix of a convolution layer . It conducts parameterization of orthogonal convolutions by adapting the construction algorithm proposed by Xiao et al . ( 2018 ) . But BCOP is slow in training . Trockman & Kolter ( 2021 ) applies the Cayley transform to a skew-symmetric convolution weight in the Fourier domain so that the convolution recovered from the Fourier domain has an orthogonal Jacobian matrix . Though it achieves higher efficiency in some applications than BCOP , it is still significantly more costly than standard convolutional layers . Skew Orthogonal Convolution ( Singla & Feizi , 2021b ) achieves orthogonal convolution through Taylor expansion of the matrix exponential . It is faster than BCOP in training but slower in evaluation . In contrast , our method is efficient in both training and evaluation . Recently , Su et al . ( 2021 ) achieves orthogonal convolutions via paraunitary systems in an elegant manner . Nevertheless , it also suffers from inefficiency issue . 3 PRELIMINARY . Notation . For a matrix M , M [ i , j ] denotes the element in the i-th row and the j-th column , M [ i , : ] denotes the i-th row and M [ : , j ] denotes the j-th column . We use the similar notation for indexing tensors . For n ∈ N , [ n ] denotes the set of n numbers from 0 to n − 1 , that is , { 0 , 1 , · · · , n − 1 } . We denote the function to obtain the singular values by σ ( · ) . Let vec ( · ) denotes the function which unfolds a matrix or higher-order tensor into a vector . Let ωk = exp ( 2πi/k ) where i = √ −1 . Let Fk ∈ Rk×k be matrix for the discrete Fourier transform ( DFT ) for k-element vectors and each entry of Fk is defined as Fk [ p , q ] = ω pq k . 3.1 RELATION BETWEEN THE JACOBIAN MATRIX AND THE WEIGHT MATRIX . We denote the input tensor of a convolution layer by X ∈ Rc×n×n , where n is the spatial size , and c is the number of input channels . We denote the convolution operation on X by convŴ ( X ) , where Ŵ ∈ Rk×k×c×c size where k is the size of the reception field and k is normally smaller than size of the input image , n. In this definition , we set the number of output channels identical to that of input channels . Meanwhile , we set the convolution stride as 1 and adapt the cyclic padding to make the output of the same size as the input . We denote the output tensor by Y ∈ Rc×n×n and each element of Y is obtained by ∀l , s , t ∈ [ c ] × [ n ] × [ n ] , Y [ l , s , t ] = ∑ r∈ [ c ] ∑ p∈ [ k ] ∑ q∈ [ k ] X [ l + r , s+ p , t+ q ] Ŵ [ p , q , l , r ] . ( 1 ) Since the convolution operation is a linear transformation , there exists a linear transformation M ∈ Rn2c×n2c which satisfies : Y = convŴ ( X ) ⇔ vec ( Y ) = Mvec ( X ) . ( 2 ) To simplify illustration , Ŵ ∈ Rk×k×c×c is normally expanded into W ∈ Rn×n×c×c through zeropadding . We term W as the expanded convolution kernel . In W , only k2c2 elements are non-zero . Based on the above definition , we review the following theorem in Sedghi et al . ( 2019 ) : Theorem 1 ( see Sedghi et al . ( 2019 ) Section 2.2 ) . For any expanded convolution kernel W ∈ Rn×n×c×c , let M is the matrix encoding the linear transformation of a convolution with W . For each p , q ∈ [ n ] × [ n ] , let P ( p , q ) be the c× c matrix computed by ∀s , t ∈ [ c ] × [ c ] , P ( p , q ) [ s , t ] = ( F > nW [ : , : , s , t ] Fn ) [ p , q ] . ( 3 ) Then σ ( M ) = ⋃ p∈ [ n ] , q∈ [ n ] σ ( P ( p , q ) ) . ( 4 ) The above theorem gives the relation between the singular values of the transformation matrix M and the expanded convolution W . Theorem 2 . For any W ∈ Rn×n×c×c , the Jacobian matrix of the convolution layer , J , is orthogonal if and only if ∀p , q ∈ [ n ] × [ n ] , each singular value of the matrix P ( p , q ) is 1 . Proof . A real matrix is orthogonal if and only if each singular value is 1 . Meanwhile , the Jacobian matrix J = ∂vec ( Y ) ∂vec ( X ) = M. Based on Theorem 1 , each singular value of M is 1 if and only if ∀p , q ∈ [ n ] × [ n ] , each singular value of the matrix P ( p , q ) is 1 , thus we complete the proof .

Summary: This paper presents a method for enforcing strict orthogonality for convolutional layers. The method is based on considering the spectral domain, where the orthogonality of the 4D conv kernel (in spatial domain) is characterized as the orthogonality of 2D matrices, for which orthogonality can be enforced by existing techniques such as Carley transform. It is shown in experiments that this method is significantly faster than the recent skew orthogonal convolution (at ICML'21) method.