Scaling-up Diverse Orthogonal Convolutional Networks by a Paraunitary Framework

1 INTRODUCTION . Convolutional neural networks ( CNNs ) , whose deployment has witnessed extensive empirical success , still exhibit a range of limitations that are not thoroughly studied . Firstly , deep convolutional networks are in general difficult to learn , and their high performance heavily relies on techniques that are not fully understood , such as skip-connections ( He et al. , 2016a ) , batch normalization ( Ioffe & Szegedy , 2015 ) , specialized initialization ( Glorot & Bengio , 2010 ) . Secondly , they are notoriously sensitive to imperceptible perturbations , including adversarial attacks ( Goodfellow et al. , 2014 ) or geometric transformations ( Azulay & Weiss , 2019 ) . Finally , a precise characterization of their generalization property is still under active investigation ( Neyshabur et al. , 2018 ; Jia et al. , 2019 ) . Orthogonal networks , which have a “ flat ” spectrum with all singular values of the layer being 1 ( each layer ’ s output norm ‖y‖ equals its input norm ‖x‖ , ∀x ) , alleviate all problems above . As shown in recent works , by enforcing orthogonality in neural nets , we obtain ( 1 ) easier optimization ( Zhang et al. , 2018a ; Qi et al. , 2020 ) : since each orthogonal layer preserves the gradient norm during backpropagation , an orthogonal network is free from gradient vanishing/ exploding problems ; ( 2 ) robustness against adversarial perturbation ( Anil et al. , 2019 ; Li et al. , 2019b ; Trockman & Kolter , 2021 ) : since each orthogonal layer is 1-Lipschitz , an orthogonal network can not amplify any perturbation to the input to flip the output prediction ; ( 3 ) better generalizability as proved in ( Jia et al. , 2019 ) : a network ’ s generalization error is positively related to the standard deviation of each linear layer ’ s singular values , thus encouraging orthogonality in the network lowers its generalization error . Despite the benefits , enforcing orthogonality in convolutional networks is challenging . To avoid the strict constraint , orthogonal initialization ( dynamical isometry ) ( Pennington et al. , 2017 ; Xiao et al. , 2018 ) and orthogonal regularization ( Wang et al. , 2019 ; Qi et al. , 2020 ) are adopted to address the gradient vanishing/exploding problems . However , these methods are not suitable for applications that require strict Lipschitzness , such as adversarial robustness ( Anil et al. , 2019 ) and residual flows ( Chen et al. , 2019 ) , as they do not enforce strict orthogonality ( and Lipschitzness ) during training . Our goal is to enforce exact orthogonality in state-of-the-art convolutional networks without expensive computations . We identify three main challenges in doing so . Challenge I : Achieving exact orthogonality throughout the training process . Prior works such as orthogonal regularization ( Wang et al. , 2019 ) and reshaped kernel orthogonality ( Jia et al. , 2017 ; Cisse et al. , 2017 ) , while enjoying algorithmic simplicity , do not meet the requirement of exact orthogonality . Note that enforcing orthogonality during training is necessary as a post-training orthogonalization of the weights can ruin the performance . Challenge II : Avoiding expensive computations . An efficient algorithm is crucial for scalability to large networks . Existing work based on projected gradient descent ( Sedghi et al. , 2019 ) , however , requires expensive projection after each update . For instance , the projection step in Sedghi et al . ( 2019 ) computes an SVD and flattens the spectrum to enforce orthogonality , which costs O ( size ( feature ) · channels3 ) for a convolutional layer . Challenge III : Scaling-up to state-of-the-art deep convolutional networks . There are many variants to the standard convolutional layer , including dilated , strided , group convolutions , which are essential for state-of-the-art deep convolutional networks . However , none of the existing methods proposes mechanisms to orthogonalize these variants . The lack of techniques , as a result , limits the broad applications of orthogonal convolutional layers to state-of-the-art deeper convolutional networks . We resolve challenges I , II , and III by proposing complete parameterizations for orthogonal 1Dconvolutions and separable orthogonal 2D-convolutions . First , using the convolution theorem ( Oppenheim et al. , 1996 ) ( convolution in the spatial domain is equivalent to multiplication in the spectral domain ) , we reduce the problem of designing orthogonal convolutions to constructing unitary matrices for all frequencies , i.e. , a paraunitary system ( Vaidyanathan , 1993 ) . Therefore , we obtain a parameterization for all separable orthogonal 2D-convolutions , guaranteeing exact orthogonality and high expressive power . Note that there is no previous approach that achieves exact orthogonality ( up to machine precision ) . For the first time , our versatile framework enables the study in the designs of deep orthogonal networks , such as skip connection , initialization , stride , and dilation . Consequently , we scale orthogonal networks to deep architectures , including ResNet and ShuffleNet , substantially outperforming their shallower counterparts . Our proposed method serves as gaining insight into the trade-off between orthogonality and expressiveness . We observe that exact orthogonality is essential in these deeper architectures with more than 10 layers ( detailed discussion of exact orthogonality in Appendix E.3 ) . Finally , we show how to deploy our orthogonal networks in Residual Flow ( Chen et al. , 2019 ) , a flow-based generative model that requires strict Lipschitzness ( Appendix F ) . Summary of Contributions : . 1 . We establish the equivalence between orthogonal convolutions in the spatial domain and paraunitary systems in the spectral domain . Consequently , we can interpret existing approaches for orthogonal convolutions as implicit designs of paraunitary systems . 2 . Based on a complete factorization of 1D paraunitary systems , we propose the first exact and complete design for orthogonal 1D-convolutions as well as separable orthogonal 2D-convolutions , ensuring exact orthogonality and high expressive power . None of the existing works achieve a complete parameterization of all orthogonal 2D-convolutions . 3 . We prove that orthogonality for various convolutional layers ( strided , dilated , group ) are also entirely characterized by paraunitary systems . Consequently , our design easily extends to these variants , ensuring both completeness and exactness of the orthogonal convolutions . 4 . We study the design considerations for orthogonal networks ( choices of skip connection , initialization , depth , width , and kernel size ) , and show that orthogonal networks can scale to deep architectures including ResNet , WideResNet , and ShuffleNet . 2 ORTHOGONAL CONVOLUTIONS VIA PARAUNITARY SYSTEMS . Designing orthogonal convolutional layer { ht , s : yt = ht , s ∗ xs } T , St=1 , s=1 ( s , t index input/output channels ) in the spatial domain is challenging . When a convolution layer is written as a matrix-vector product , where the matrix is block-circulant and its ( t , s ) th takes a circulant form Cir ( ht , s ) as : Cir ( ht , s ) = ht , s [ 1 ] ht , s [ N ] · · · ht , s [ 2 ] ht , s [ 2 ] ht , s [ 1 ] ht , s [ N ] · · · ... . . . . . . ... ht , s [ N ] · · · ht , s [ 2 ] ht , s [ 1 ] ∈ RN×N . ( 2.1 ) Therefore , the layer is orthogonal if the block-circulant matrix [ Cir ( ht , s ) ] T , S t=1 , s=1 is orthogonal . However , it is not obvious how to enforce orthogonality in a block-circulant matrix . 2.1 ACHIEVING ORTHOGONAL CONVOLUTIONS BY PARAUNITARY SYSTEMS . We propose a novel design of orthogonal convolutions from a spectral domain perspective , motivated by the convolution theorem ( Theorem 2.1 ) . For simplicity , we group the entries at the same locations into a vector/matrix , e.g. , we denote { xs [ n ] } Ss=1 as x [ n ] ∈ RS and { ht , s [ n ] } T , S t=1 , s=1 as h [ n ] ∈ RT×S . Theorem 2.1 ( Convolution theorem ( Oppenheim et al. , 1996 ) ) . For a standard convolution layer h : y [ i ] = ∑ n h [ n ] x [ i − n ] , the convolution in the spatial domain is equivalent to a matrixvector multiplication in the spectral domain , i.e. , Y ( z ) = H ( z ) X ( z ) , ∀z ∈ C. Here , X ( z ) =∑N−1 n=0 x [ n ] z −n , Y ( z ) = ∑N−1 n=0 y [ n ] z −n , H ( z ) = ∑L n=−L h [ n ] z −n denote the z-transforms of input , output , kernel respectively , where N is the length of x , y and [ −L , L ] is the span of the filter h. The convolution theorem states that a standard convolution layer is a matrix-vector multiplication in the spectral domain . As long as the transfer matrix H ( z ) is unitary at z = ejω for all frequencies ∀ω ∈ R ( j is the imaginary unit ) , the convolutional layer h is orthogonal . Therefore , as a major novelty of our paper , we design orthogonal convolutions through designing unitary transfer matrix H ( ejω ) at all frequencies ω ∈ R , which is known as a paraunitary systems ( Vaidyanathan , 1993 ; Strang & Nguyen , 1996 ) . We prove in Theorem B.5 that a convolutional layer is orthogonal in the spatial domain if and only if it is paraunitary in the spectral domain . Benefits through paraunitary systems . ( 1 ) The spectral representation simplifies the designs of orthogonal convolutions and avoids the analysis of block-circulant structure . ( 2 ) Since paraunitary systems are necessary and sufficient for orthogonal convolutions , it is impossible to find an orthogonal convolution whose transfer matrix is not paraunitary . ( 3 ) There exists a complete factorization of paraunitary systems : any paraunitaryH ( z ) can be realized through multiplications in the spectral domain , as will show in Equation ( 2.2a ) . ( 4 ) Since multiplications in spectral domain correspond to convolutions in spatial domain , any orthogonal convolution can be realized as cascaded convolutions of multiple sub-layers , each parameterized by an orthogonal matrix . ( 5 ) There are mature methods that represent orthogonal matrices via unconstrained parameters . Consequently , we can learn orthogonal convolutions using standard optimizers on a model parameterized via our design . Interpretation of existing methods . Since paraunitary system is a necessary and sufficient condition for orthogonal convolution , all existing approaches , including singular value clipping and masking ( SVCM ) ( Sedghi et al. , 2019 ) , block convolution orthogonal parameterization ( BCOP ) ( Li et al. , 2019b ) , Cayley Convolution ( CayleyConv ) ( Trockman & Kolter , 2021 ) , skew orthogonal convolution ( SOC ) design paraunitary systems implicitly . We discuss these interpretations in Appendix C.3.3 , and show that our SC-Fac has the lowest computational complexity among all approaches . 2.2 REALIZING PARAUNITARY SYSTEMS VIA RE-PARAMETERIZATION . After reducing the problem of orthogonal convolutions to paraunitary systems , we are left with the question of how to realize paraunitary systems . We use a complete factorization form of paraunitary systems to realize any paraunitary systems . According to Theorem C.7 , any paraunitary systemH ( z ) can be written as the form in Equation ( 2.2a ) and anyH ( z ) in this form is a paraunitary system : H ( z ) = V ( z ; U ( −L ) ) · · ·V ( z ; U ( −1 ) ) QV ( z−1 ; U ( 1 ) ) · · ·V ( z−1 ; U ( L ) ) , ( 2.2a ) where V ( z ; U ( ` ) ) = ( I −U ( ` ) U ( ` ) > ) +U ( ` ) U ( ` ) > z , ∀ ` ∈ { −L , · · · , −1 } ∪ { 1 , · · · , L } . ( 2.2b ) Here Q is an orthogonal matrix and each U ( ` ) is a column-orthogonal matrix whose number of columns is sampled uniformly from { 1 , · · · , T } . As multiplications in the spectral domain are equivalent to convolutions in the spatial domain , this complete spectral factorization of paraunitary systems in Equation ( 2.2a ) allows us to parameterize any orthogonal convolution in the spatial domain as cascaded convolutions of V ( z ; U ( ` ) ) ’ s spatial counterparts and the orthogonal matrixQ . Model design in the spatial domain . We obtain a complete design of orthogonal 1D-convolutions : using learnable ( column ) -orthogonal matrices ( { U ( ` ) } −1 ` =−L , Q , { U ( ` ) } L ` =1 ) , we parameterize a size ( L+ L+ 1 ) convolution as cascaded convolutions of the following filters in the spatial domain { [ I −U ( ` ) U ( ` ) > , U ( ` ) U ( ` ) > ] } −1 ` =−L , Q , { [ U ( ` ) U ( ` ) > , I −U ( ` ) U ( ` ) > ] } L ` =1 . ( 2.3 ) Figure 1 provides a visualization of our proposed design of orthogonal convolution layers ; each block denotes a convolution and the form of the filter is displayed in each of the block . In practice , we compose all ( L+ L+ 1 ) filters into one for orthogonal convolution , which not only increases the computational parallelism but also avoids storing intermediate outputs between filters . Using this 1D-convolution , we obtain a complete design for separable orthogonal 2D-convolutions , which can be represented as a convolution of two orthogonal 1D-convolutional layers . With separability , we obtain a design of orthogonal 2D-convolutions by composing two complete designs of orthogonal 1D-convolutions . As a result , we can parameterize separable orthogonal 2D-convolution with filter size ( L1+L1+1 ) × ( L2+L2+1 ) as a convolution of two orthogonal 1D-convolutions with learnable ( column- ) orthogonal matrices ( { U ( ` ) 1 } −1 ` =−L1 , Q1 , { U ( ` ) 1 } L1 ` =1 ) and ( { U ( ` ) 2 } −1 ` =−L2 , Q2 , { U ( ` ) 2 } L2 ` =1 ) . With a complete factorization of paraunitary systems ( 1D and separable 2D ) , we reduce the problem of designing orthogonal convolutions to the one for orthogonal matrices . Parameterization for orthogonal matrices . In Appendix C.4 , we perform a comparative study on different parameterizations of orthogonal matrices , including the Björck orthogonalization ( Anil et al. , 2019 ; Li et al. , 2019b ) , the Cayley transform ( Helfrich et al. , 2018 ; Maduranga et al. , 2019 ) , and the exponential map ( Lezcano-Casado & Martínez-Rubio , 2019 ) . In our implementation , we adopt a modified exponential map due to its efficiency , exactness , and completeness . The exponential map is a surjective mapping from a skew-symmetry matrixA to a special orthogonal matrix U ( i.e. , det ( U ) = 1 ) with U = exp ( A ) = I +A+A2/2 + · · · , where the infinite series is computed up to machine-precision ( Higham , 2009 ) . To parameterize all orthogonal matrices , we introduce an additional orthogonal matrixQ in U = Q exp ( A ) , whereQ is ( randomly ) generated at initialization and fixed during training ( Lezcano Casado , 2019 ) . Now we have an end-to-end pipeline for orthogonal convolutions as shown in Figure 2 ( See Algorithm 1 in Appendix E for a pseudo implementation ) . Since our method relies on separability and complete factorization for 1D paraunitary systems , we call it Separable Complete Factorization ( SC-Fac ) .

Summary: This paper proposed a theoretical framework for orthogonal convolutional layers based on the equivalency of orthogonal convolution in the spatial domain and the paraunitary systems in the spectral domain. The proposed method parametrizes the orthogonal convolution layer as compositions of multiple convolutions in the spatial domain, resulting in exact orthogonality. The layers are also more memory and computationally efficient than most previous methods.