Scaling-up Diverse Orthogonal Convolutional Networks by a Paraunitary Framework

1 INTRODUCTION . Convolutional neural networks ( CNNs ) , whose deployment has witnessed extensive empirical success , still exhibit a range of limitations that are not thoroughly studied . Firstly , deep convolutional networks are in general difficult to learn , and their high performance heavily relies on techniques that are not fully understood , such as skip-connections ( He et al. , 2016a ) , batch normalization ( Ioffe & Szegedy , 2015 ) , specialized initialization ( Glorot & Bengio , 2010 ) . Secondly , they are notoriously sensitive to imperceptible perturbations , including adversarial attacks ( Goodfellow et al. , 2014 ) or geometric transformations ( Azulay & Weiss , 2019 ) . Finally , a precise characterization of their generalization property is still under active investigation ( Neyshabur et al. , 2018 ; Jia et al. , 2019 ) . Orthogonal networks , which have a “ flat ” spectrum with all singular values of the layer being 1 ( each layer ’ s output norm ‖y‖ equals its input norm ‖x‖ , ∀x ) , alleviate all problems above . As shown in recent works , by enforcing orthogonality in neural nets , we obtain ( 1 ) easier optimization ( Zhang et al. , 2018a ; Qi et al. , 2020 ) : since each orthogonal layer preserves the gradient norm during backpropagation , an orthogonal network is free from gradient vanishing/ exploding problems ; ( 2 ) robustness against adversarial perturbation ( Anil et al. , 2019 ; Li et al. , 2019b ; Trockman & Kolter , 2021 ) : since each orthogonal layer is 1-Lipschitz , an orthogonal network can not amplify any perturbation to the input to flip the output prediction ; ( 3 ) better generalizability as proved in ( Jia et al. , 2019 ) : a network ’ s generalization error is positively related to the standard deviation of each linear layer ’ s singular values , thus encouraging orthogonality in the network lowers its generalization error . Despite the benefits , enforcing orthogonality in convolutional networks is challenging . To avoid the strict constraint , orthogonal initialization ( dynamical isometry ) ( Pennington et al. , 2017 ; Xiao et al. , 2018 ) and orthogonal regularization ( Wang et al. , 2019 ; Qi et al. , 2020 ) are adopted to address the gradient vanishing/exploding problems . However , these methods are not suitable for applications that require strict Lipschitzness , such as adversarial robustness ( Anil et al. , 2019 ) and residual flows ( Chen et al. , 2019 ) , as they do not enforce strict orthogonality ( and Lipschitzness ) during training . Our goal is to enforce exact orthogonality in state-of-the-art convolutional networks without expensive computations . We identify three main challenges in doing so . Challenge I : Achieving exact orthogonality throughout the training process . Prior works such as orthogonal regularization ( Wang et al. , 2019 ) and reshaped kernel orthogonality ( Jia et al. , 2017 ; Cisse et al. , 2017 ) , while enjoying algorithmic simplicity , do not meet the requirement of exact orthogonality . Note that enforcing orthogonality during training is necessary as a post-training orthogonalization of the weights can ruin the performance . Challenge II : Avoiding expensive computations . An efficient algorithm is crucial for scalability to large networks . Existing work based on projected gradient descent ( Sedghi et al. , 2019 ) , however , requires expensive projection after each update . For instance , the projection step in Sedghi et al . ( 2019 ) computes an SVD and flattens the spectrum to enforce orthogonality , which costs O ( size ( feature ) · channels3 ) for a convolutional layer . Challenge III : Scaling-up to state-of-the-art deep convolutional networks . There are many variants to the standard convolutional layer , including dilated , strided , group convolutions , which are essential for state-of-the-art deep convolutional networks . However , none of the existing methods proposes mechanisms to orthogonalize these variants . The lack of techniques , as a result , limits the broad applications of orthogonal convolutional layers to state-of-the-art deeper convolutional networks . We resolve challenges I , II , and III by proposing complete parameterizations for orthogonal 1Dconvolutions and separable orthogonal 2D-convolutions . First , using the convolution theorem ( Oppenheim et al. , 1996 ) ( convolution in the spatial domain is equivalent to multiplication in the spectral domain ) , we reduce the problem of designing orthogonal convolutions to constructing unitary matrices for all frequencies , i.e. , a paraunitary system ( Vaidyanathan , 1993 ) . Therefore , we obtain a parameterization for all separable orthogonal 2D-convolutions , guaranteeing exact orthogonality and high expressive power . Note that there is no previous approach that achieves exact orthogonality ( up to machine precision ) . For the first time , our versatile framework enables the study in the designs of deep orthogonal networks , such as skip connection , initialization , stride , and dilation . Consequently , we scale orthogonal networks to deep architectures , including ResNet and ShuffleNet , substantially outperforming their shallower counterparts . Our proposed method serves as gaining insight into the trade-off between orthogonality and expressiveness . We observe that exact orthogonality is essential in these deeper architectures with more than 10 layers ( detailed discussion of exact orthogonality in Appendix E.3 ) . Finally , we show how to deploy our orthogonal networks in Residual Flow ( Chen et al. , 2019 ) , a flow-based generative model that requires strict Lipschitzness ( Appendix F ) . Summary of Contributions : . 1 . We establish the equivalence between orthogonal convolutions in the spatial domain and paraunitary systems in the spectral domain . Consequently , we can interpret existing approaches for orthogonal convolutions as implicit designs of paraunitary systems . 2 . Based on a complete factorization of 1D paraunitary systems , we propose the first exact and complete design for orthogonal 1D-convolutions as well as separable orthogonal 2D-convolutions , ensuring exact orthogonality and high expressive power . None of the existing works achieve a complete parameterization of all orthogonal 2D-convolutions . 3 . We prove that orthogonality for various convolutional layers ( strided , dilated , group ) are also entirely characterized by paraunitary systems . Consequently , our design easily extends to these variants , ensuring both completeness and exactness of the orthogonal convolutions . 4 . We study the design considerations for orthogonal networks ( choices of skip connection , initialization , depth , width , and kernel size ) , and show that orthogonal networks can scale to deep architectures including ResNet , WideResNet , and ShuffleNet . 2 ORTHOGONAL CONVOLUTIONS VIA PARAUNITARY SYSTEMS . Designing orthogonal convolutional layer { ht , s : yt = ht , s ∗ xs } T , St=1 , s=1 ( s , t index input/output channels ) in the spatial domain is challenging . When a convolution layer is written as a matrix-vector product , where the matrix is block-circulant and its ( t , s ) th takes a circulant form Cir ( ht , s ) as : Cir ( ht , s ) = ht , s [ 1 ] ht , s [ N ] · · · ht , s [ 2 ] ht , s [ 2 ] ht , s [ 1 ] ht , s [ N ] · · · ... . . . . . . ... ht , s [ N ] · · · ht , s [ 2 ] ht , s [ 1 ] ∈ RN×N . ( 2.1 ) Therefore , the layer is orthogonal if the block-circulant matrix [ Cir ( ht , s ) ] T , S t=1 , s=1 is orthogonal . However , it is not obvious how to enforce orthogonality in a block-circulant matrix . 2.1 ACHIEVING ORTHOGONAL CONVOLUTIONS BY PARAUNITARY SYSTEMS . We propose a novel design of orthogonal convolutions from a spectral domain perspective , motivated by the convolution theorem ( Theorem 2.1 ) . For simplicity , we group the entries at the same locations into a vector/matrix , e.g. , we denote { xs [ n ] } Ss=1 as x [ n ] ∈ RS and { ht , s [ n ] } T , S t=1 , s=1 as h [ n ] ∈ RT×S . Theorem 2.1 ( Convolution theorem ( Oppenheim et al. , 1996 ) ) . For a standard convolution layer h : y [ i ] = ∑ n h [ n ] x [ i − n ] , the convolution in the spatial domain is equivalent to a matrixvector multiplication in the spectral domain , i.e. , Y ( z ) = H ( z ) X ( z ) , ∀z ∈ C. Here , X ( z ) =∑N−1 n=0 x [ n ] z −n , Y ( z ) = ∑N−1 n=0 y [ n ] z −n , H ( z ) = ∑L n=−L h [ n ] z −n denote the z-transforms of input , output , kernel respectively , where N is the length of x , y and [ −L , L ] is the span of the filter h. The convolution theorem states that a standard convolution layer is a matrix-vector multiplication in the spectral domain . As long as the transfer matrix H ( z ) is unitary at z = ejω for all frequencies ∀ω ∈ R ( j is the imaginary unit ) , the convolutional layer h is orthogonal . Therefore , as a major novelty of our paper , we design orthogonal convolutions through designing unitary transfer matrix H ( ejω ) at all frequencies ω ∈ R , which is known as a paraunitary systems ( Vaidyanathan , 1993 ; Strang & Nguyen , 1996 ) . We prove in Theorem B.5 that a convolutional layer is orthogonal in the spatial domain if and only if it is paraunitary in the spectral domain . Benefits through paraunitary systems . ( 1 ) The spectral representation simplifies the designs of orthogonal convolutions and avoids the analysis of block-circulant structure . ( 2 ) Since paraunitary systems are necessary and sufficient for orthogonal convolutions , it is impossible to find an orthogonal convolution whose transfer matrix is not paraunitary . ( 3 ) There exists a complete factorization of paraunitary systems : any paraunitaryH ( z ) can be realized through multiplications in the spectral domain , as will show in Equation ( 2.2a ) . ( 4 ) Since multiplications in spectral domain correspond to convolutions in spatial domain , any orthogonal convolution can be realized as cascaded convolutions of multiple sub-layers , each parameterized by an orthogonal matrix . ( 5 ) There are mature methods that represent orthogonal matrices via unconstrained parameters . Consequently , we can learn orthogonal convolutions using standard optimizers on a model parameterized via our design . Interpretation of existing methods . Since paraunitary system is a necessary and sufficient condition for orthogonal convolution , all existing approaches , including singular value clipping and masking ( SVCM ) ( Sedghi et al. , 2019 ) , block convolution orthogonal parameterization ( BCOP ) ( Li et al. , 2019b ) , Cayley Convolution ( CayleyConv ) ( Trockman & Kolter , 2021 ) , skew orthogonal convolution ( SOC ) design paraunitary systems implicitly . We discuss these interpretations in Appendix C.3.3 , and show that our SC-Fac has the lowest computational complexity among all approaches . 2.2 REALIZING PARAUNITARY SYSTEMS VIA RE-PARAMETERIZATION . After reducing the problem of orthogonal convolutions to paraunitary systems , we are left with the question of how to realize paraunitary systems . We use a complete factorization form of paraunitary systems to realize any paraunitary systems . According to Theorem C.7 , any paraunitary systemH ( z ) can be written as the form in Equation ( 2.2a ) and anyH ( z ) in this form is a paraunitary system : H ( z ) = V ( z ; U ( −L ) ) · · ·V ( z ; U ( −1 ) ) QV ( z−1 ; U ( 1 ) ) · · ·V ( z−1 ; U ( L ) ) , ( 2.2a ) where V ( z ; U ( ` ) ) = ( I −U ( ` ) U ( ` ) > ) +U ( ` ) U ( ` ) > z , ∀ ` ∈ { −L , · · · , −1 } ∪ { 1 , · · · , L } . ( 2.2b ) Here Q is an orthogonal matrix and each U ( ` ) is a column-orthogonal matrix whose number of columns is sampled uniformly from { 1 , · · · , T } . As multiplications in the spectral domain are equivalent to convolutions in the spatial domain , this complete spectral factorization of paraunitary systems in Equation ( 2.2a ) allows us to parameterize any orthogonal convolution in the spatial domain as cascaded convolutions of V ( z ; U ( ` ) ) ’ s spatial counterparts and the orthogonal matrixQ . Model design in the spatial domain . We obtain a complete design of orthogonal 1D-convolutions : using learnable ( column ) -orthogonal matrices ( { U ( ` ) } −1 ` =−L , Q , { U ( ` ) } L ` =1 ) , we parameterize a size ( L+ L+ 1 ) convolution as cascaded convolutions of the following filters in the spatial domain { [ I −U ( ` ) U ( ` ) > , U ( ` ) U ( ` ) > ] } −1 ` =−L , Q , { [ U ( ` ) U ( ` ) > , I −U ( ` ) U ( ` ) > ] } L ` =1 . ( 2.3 ) Figure 1 provides a visualization of our proposed design of orthogonal convolution layers ; each block denotes a convolution and the form of the filter is displayed in each of the block . In practice , we compose all ( L+ L+ 1 ) filters into one for orthogonal convolution , which not only increases the computational parallelism but also avoids storing intermediate outputs between filters . Using this 1D-convolution , we obtain a complete design for separable orthogonal 2D-convolutions , which can be represented as a convolution of two orthogonal 1D-convolutional layers . With separability , we obtain a design of orthogonal 2D-convolutions by composing two complete designs of orthogonal 1D-convolutions . As a result , we can parameterize separable orthogonal 2D-convolution with filter size ( L1+L1+1 ) × ( L2+L2+1 ) as a convolution of two orthogonal 1D-convolutions with learnable ( column- ) orthogonal matrices ( { U ( ` ) 1 } −1 ` =−L1 , Q1 , { U ( ` ) 1 } L1 ` =1 ) and ( { U ( ` ) 2 } −1 ` =−L2 , Q2 , { U ( ` ) 2 } L2 ` =1 ) . With a complete factorization of paraunitary systems ( 1D and separable 2D ) , we reduce the problem of designing orthogonal convolutions to the one for orthogonal matrices . Parameterization for orthogonal matrices . In Appendix C.4 , we perform a comparative study on different parameterizations of orthogonal matrices , including the Björck orthogonalization ( Anil et al. , 2019 ; Li et al. , 2019b ) , the Cayley transform ( Helfrich et al. , 2018 ; Maduranga et al. , 2019 ) , and the exponential map ( Lezcano-Casado & Martínez-Rubio , 2019 ) . In our implementation , we adopt a modified exponential map due to its efficiency , exactness , and completeness . The exponential map is a surjective mapping from a skew-symmetry matrixA to a special orthogonal matrix U ( i.e. , det ( U ) = 1 ) with U = exp ( A ) = I +A+A2/2 + · · · , where the infinite series is computed up to machine-precision ( Higham , 2009 ) . To parameterize all orthogonal matrices , we introduce an additional orthogonal matrixQ in U = Q exp ( A ) , whereQ is ( randomly ) generated at initialization and fixed during training ( Lezcano Casado , 2019 ) . Now we have an end-to-end pipeline for orthogonal convolutions as shown in Figure 2 ( See Algorithm 1 in Appendix E for a pseudo implementation ) . Since our method relies on separability and complete factorization for 1D paraunitary systems , we call it Separable Complete Factorization ( SC-Fac ) .

Summary: This paper suggests a framework for designing orthogonal convolutions using paraunitary systems. The proposed framework studies orthogonalization of different convolution operations. The proposed methods were examined on several datasets in comparison with state-of-the-art orthogonalization methods.