RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks

1 INTRODUCTION . While deep convolutional neural networks ( CNN ) have been widely used in computer vision and image processing applications , they are not designed to handle large group actions like rotations , which degrade the performance of CNN in many tasks ( Cheng et al. , 2016 ; Hallman & Fowlkes , 2015 ; Jaderberg et al. , 2015b ; Laptev et al. , 2016 ; Maninis et al. , 2016 ) . The regular convolutional layer is equivariant to input translations , but not other group actions . An indirect way to encode group information into the deep representation is to conduct generalized convolutions across the group as well , as in Cohen & Welling ( 2016a ) . In theory , this approach can guarantee the group equivariance of the learned representations , which provides better interpretability and regularity as well as the capability of estimating the group action in localization , boundary detection , etc . For the important case of 2D rotations , group-equivariant CNNs have been constructed in several recent works , e.g. , ( Weiler et al. , 2017 ) , Harmonic Net ( Worrall et al. , 2017 ) and Oriented Response Net ( Zhou et al. , 2017 ) . In such networks , the layer-wise output has an extra index representing the group element ( c.f . Table 1 ) , and consequently , the convolution must be across the space and the group jointly ( proved in Section 3.1 ) . This typically incurs a significant increase in the number of parameters and computational load , even with the adoption of steerable filters ( Freeman et al. , 1991 ; Weiler et al. , 2017 ; Worrall et al. , 2017 ) . In parallel , low-rank factorized filters have been proposed for sparse coding as well as the compression and regularization of deep networks . In particular , Qiu et al . ( 2018 ) showed that decomposing filters under non-adaptive bases can be an effective way to reduce the model size of CNNs without sacrificing performance . However , these approaches do not directly apply to be group-equivariant . We review these connections in more detail in Section 1.1 . This paper proposes a truncated bases decomposition of the filters in group-equivariant CNNs , which we call the rotation-equivariant CNN with decomposed convolutional filters ( RotDCF ) . Since we need a joint convolution over R2 and SO ( 2 ) , the bases are also joint across the two geometrical domains , c.f . Figure 1 . The benefits of bases decomposition are three-fold : ( 1 ) Reduction of the number of parameters and computational complexity of rotation-equivariant CNNs , c.f . Section 2.3 ; ( 2 ) Implicit regularization of the convolutional filters , leading to improved robustness of the learned deep representation shown experimentally in Section 4 ; ( 3 ) Theoretical guarantees on stability of the equivariant representation to input deformations , which follow from a generic condition on the filters in the decomposed form , c.f . Section 3.2 . 1.1 RELATED WORK . Learning with factorized filters . In the sparse coding literature , Rubinstein et al . ( 2010 ) proposed the factorization of learned dictionaries under another prefixed dictionary . Separable filters were used in Rigamonti et al . ( 2013 ) to learn the coding of images . Papyan et al . ( 2017 ) interpreted CNN as an iterated convolutional sparse coding machine , and in this view , the factorized filters should correspond to a “ dictionary of the dictionary ” as in Rubinstein et al . ( 2010 ) . In the deep learning literature , low-rank factorization of convolutional filters has been previously used to remove redundancy in trained CNNs ( Denton et al. , 2014 ; Jaderberg et al. , 2014 ) . The compression of deep networks has also been studied in Chen et al . ( 2015 ) ; Han et al . ( 2016 ; 2015 ) , SqueezeNet ( Iandola et al. , 2016 ) , etc. , where the low-rank factorization of filters can be utilized . MobileNets ( Howard et al. , 2017 ) used depth-wise separable convolutions to obtain significant compression . Tensor decomposition of convolutional layers was used in Lebedev et al . ( 2014 ) for CPU speedup . Tai et al . ( 2015 ) proposed low-rank-regularized filters and obtained improved classification accuracy with reduced computation . Qiu et al . ( 2018 ) studied decomposed-filter CNN with prefixed bases and trainable expansion coefficients , showing that the truncated bases decomposition incurs almost no decrease in classification accuracy while significantly reducing the model size and improving the robustness of the deep features . None of the above networks are group equivariant . Group-equivariant deep networks . The encoding of group information into network representations has been studied extensively . Among earlier works , transforming auto-encoders ( Hinton et al. , 2011 ) used a non-convolutional network to learn group-invariant features and compared with handcrafted ones . Rotation-invariant descriptors were studied in Schmidt & Roth ( 2012b ) with product models , and in Jaderberg et al . ( 2015a ) ; Kivinen & Williams ( 2011 ) ; Schmidt & Roth ( 2012a ) by estimating the specific image transformation . Gonzalez et al . ( 2016 ) ; Wu et al . ( 2015 ) proposed rotating conventional filters to perform rotation-invariant texture and image classification . The joint convolution across space and rotation has been studied in the scattering transform ( Oyallon & Mallat , 2015 ; Sifre & Mallat , 2013 ) . Group-equivariant CNN was considered by Cohen & Welling ( 2016a ) , which handled several finite small-order discrete groups on the input image . Rotation-equivariant CNN was later developed in Weiler et al . ( 2017 ) ; Worrall et al . ( 2017 ) ; Zhou et al . ( 2017 ) and elsewhere . In particular , steerable filters were used in Cohen & Welling ( 2016b ) ; Weiler et al . ( 2017 ) ; Worrall et al . ( 2017 ) . SO ( 3 ) -equivariant CNN for signals on spheres was studied in Cohen et al . ( 2018 ) in a different setting . Overall , the efficiency of equivariant CNNs remains to be improved since the model is typically several times larger than that of a regular CNN . The current paper adopts bases-decomposed filters previously studied in the non-equivariant setting ( Qiu et al. , 2018 ) , however , the joint convolution scheme in equivariant CNNs is over the two geometries of space and orientation simultaneously and neither the approach nor the analysis there can be directly applied . 2 ROTATION-EQUIVARIANT DCF NET . 2.1 ROTATION-EQUIVARIANT CNN . A rotation-equivariant CNN indexes the channels by the SO ( 2 ) group ( Weiler et al. , 2017 ; Zhou et al. , 2017 ) : The l-th layer output is written as x ( l ) ( u , α , λ ) , the position u ∈ R2 , the rotation α ∈ S1 , and λ ∈ [ Ml ] , Ml being the number of unstructured channel indices . Throughout the paper , [ m ] stands for the set { 1 , · · · , m } . We denote the group SO ( 2 ) also by the circle S1 since the former is parametrized by the rotation angle . The convolutional filter at the l-th layer is represented as W ( l ) λ′ , λ ( v , α ) , λ ′ ∈ [ Ml−1 ] , λ ∈ [ Ml ] , v ∈ R2 , α ∈ S1 , except for the 1st layer where there is no indexing of α . In practice , S1 is discretized into Nθ points on ( 0 , 2π ) . We denote the summation over u and α by continuous integration , and the notation ∫ S1 ( · · · ) dα means 12π ∫ 2π 0 ( · · · ) dα . Let the 2D rotation by angle t be denoted by Θt , in the 1st layer of the group-invariant CNN , x ( 1 ) ( u , α , λ ) = σ ( M0∑ λ′=1 ∫ S1 ∫ R2 x ( 0 ) ( u+ v′ , λ′ ) W ( 1 ) λ′ , λ ( Θαv ′ ) dv′ + b ( 1 ) ( λ ) ) . ( 1 ) Note that the 1st layer output has Nθ orientations indexed by α , forming a “ channel geometry ” ( Table 1 ) . For l > 1 , the convolution is jointly over R2 and SO ( 2 ) , which takes the form as x ( l ) ( u , α , λ ) = σ Ml−1∑ λ′=1 ∫ S1 ∫ R2 x ( l−1 ) ( u+ v′ , α′ , λ′ ) W ( l ) λ′ , λ ( Θαv ′ , α′ − α ) dv′dα′ + b ( l ) ( λ ) . ( 2 ) The joint convolution over R2 and SO ( 2 ) is both sufficient and necessary to guarantee groupequivariance ( Theorem 3.1 ) . While group equivariance is a desirable property , the model size and computation can be increased significantly due to the extra index α ∈ [ Nθ ] . 2.2 DECOMPOSED FILTERS UNDER STEERABLE BASES . We decompose the filters with respect to u and α simultaneously : Let { ψk } k be a set of bases on the unit 2D disk , and { ϕm } m be bases on S1 . At the l-th layer , let jl be the scale of the filter in u , and ψj , k = 2 −2jψk ( 2 −ju ) ( the filter is supported on the disk of radius 2jl ) . Since we use continuous convolutions , the down-sampling by “ pooling ” is modeled by the rescaling of the filters in space . The decomposed filters are of the form W ( 1 ) λ′ , λ ( v ) = ∑ k a ( 1 ) λ′ , λ ( k ) ψj1 , k ( v ) , W ( l ) λ′ , λ ( v , β ) = ∑ k ∑ m a ( l ) λ′ , λ ( k , m ) ψjl , k ( v ) ϕm ( β ) , l > 1 , ( 3 ) which is illustrated in Figure 1 ( for l > 1 ) . We use Fourier-Bessel ( FB ) bases for { ψk } k and Fourier bases for { ϕm } m. Specifically , ϕm ( α ) = eimα , and FB basis has the expression ψk ( r , θ ) = cm , qJm ( Rm , qr ) e imθ , 0 ≤ r ≤ 1 , 0 ≤ θ ≤ 2π , k = ( m , q ) ( 4 ) wherem is the angular frequency , q the radial frequency , Jm the Bessel function ( Rm , q the q-th root of Jm ) and cm , q a normalizing constant ( Abramowitz & Stegun , 1964 ) . Both bases are “ steerable ” , i.e . the operation of rotation is a diagonalized linear transform under both bases . In the complexvalued version , ψk ( Θtv ) = e−im ( k ) tψk ( v ) , ϕm ( α − t ) = e−imtϕm ( α ) . This means that after the convolutions on R2 × S1 with the bases ψk ( v ) ϕl ( α ) are computed for all k and l , both up to certain truncation , the joint convolution ( 1 ) , ( 2 ) with all rotated filters can be calculated by the algebraic manipulation of the expansion coefficients a ( l ) λ′ , λ ( k , m ) , and without any re-computation of the spatial-rotation joint convolution . Standard real-valued versions of the bases ψk and ϕm in sin ’ s and cos ’ s are used in practice . During training , only the expansion coefficients a ’ s are updated , and the bases are fixed . Apart from the saving of parameters and computation , the bases truncation also regularizes the convolutional filters by discarding the high frequency components . As a result , DCF Net reduces response to those components in the input at all layers , which improves the robustness of the learned feature without affecting recognition performance . The visibly smoother trained filters in RotDCF are shown in Figure A.1 , demonstrating the same regularization effects as in Qiu et al . ( 2018 ) , the latter being without the rotation-equivariant setting . The theoretical properties of RotDCF Net , particularly the representation stability , will be analyzed in Section 3 .

Summary: This paper combines the benefits of using joint steerable filters (using the SO(2) group) for designing rotation-equivariant CNNs with those of decomposing the filters (using Fourier-Bessel bases) for reducing the computational complexity. In addition, this leads to a compressed model and filter regularization. The authors give theoretical guarantees on the rotation equivariance and representation stability with respect to in and out of plane rotation. Empirical results show that the model attains better accuracy compared to CNNs and non-rotation-equivariant deep networks while using fewer parameters and also performs similarly to a rotation-equivariant model with much bigger capacity.