Scale-Equivariant Neural Networks with Decomposed Convolutional Filters

1 INTRODUCTION . Convolutional neural networks ( CNNs ) have achieved great success in machine learning problems such as image classification ( Krizhevsky et al. , 2012 ) , object detection ( Ren et al. , 2015 ) , and semantic segmentation ( Long et al. , 2015 ; Ronneberger et al. , 2015 ) . Compared to fully-connected networks , CNNs through spatial weight sharing have the benefit of being translation-equivariant , i.e. , translating the input leads to a translated version of the output . This property is crucial for many vision tasks such as image recognition and segmentation . However , regular CNNs are not equivariant to other important group transformations such as rescaling and rotation , and it is beneficial in some applications to also encode such group information explicitly into the network representation . Several network architectures have been designed to achieve ( 2D ) roto-translation-equivariance ( SE ( 2 ) -equivariance ) ( Cheng et al. , 2019 ; Marcos et al. , 2017 ; Weiler et al. , 2018b ; Worrall et al. , 2017 ; Zhou et al. , 2017 ) , i.e. , roughly speaking , if the input is spatially rotated and translated , the output is transformed accordingly . The feature maps of such networks typically include an extra index for the rotation group SO ( 2 ) . Building on the idea of group convolutions proposed by Cohen & Welling ( 2016 ) for discrete symmetry groups , Cheng et al . ( 2019 ) and Weiler et al . ( 2018b ) constructed SE ( 2 ) -equivariant CNNs by conducting group convolutions jointly across the space and SO ( 2 ) using steerable filters ( Freeman & Adelson , 1991 ) . Scaling-translation-equivariant ( ST -equivariant ) CNNs , on the other hand , have typically been studied in a less general setting in the existing literature ( Kanazawa et al. , 2014 ; Marcos et al. , 2018 ; Xu et al. , 2014 ; Ghosh & Gupta , 2019 ) . In particular , to the best of our knowledge , a joint convolution across the space and the scaling group S has yet been proposed to achieve equivariance in the most general form . This is possibly because of two difficulties one encounters when dealing with the scaling group : First , unlike SO ( 2 ) , it is an acyclic and unbounded group ; second , an extra index in S incurs a significant increase in model parameters and computational burden . Moreover , since the scaling transformation is rarely perfect in practice ( due to changing view angle or numerical discretization ) , one needs to quantify and promote the deformation robustness of the equivariant representation ( i.e. , is the model still “ approximately ” equivariant if the scaling transformation is “ contaminated ” by a nuisance input deformation ) , which , to the best of our knowledge , has yet been studied in prior works . The purpose of this paper is to address the aforementioned theoretical and practical issues in the construction of ST -equivariant CNN models . Specifically , our contribution is three-fold : 1 . We propose a general ST -equivariant CNN architecture with a joint convolution over R2and S , which is proved in Section 4 to be both sufficient and necessary to achieve ST -equivariance . 2 . A truncated decomposition of the convolutional filters under a pre-fixed separable basis on the two geometric domains ( R2 and S ) is used to reduce the model size and computational cost . 3 . We prove the representation stability of the proposed architecture up to equivariant scaling action of the input signal . Our contribution to the family of group-equivariant CNNs is non-trivial ; in particular , the scaling group unlike SO ( 2 ) is acyclic and non-compact . This poses challenges both in theory and in practice , so that many previous works on group-equivariant CNNs can not be directly extended . We introduce new algorithm design and mathematical techniques to obtain the first general ST -equivariant CNN in literature with both computational efficiency and proved representation stability . 2 RELATED WORK . Mixed-scale and ST -equivariant CNNs . Incorporating multiscale information into a CNN representation has been studied in many existing works . The Inception net ( Szegedy et al. , 2015 ) and its generalizations ( Szegedy et al. , 2017 ; 2016 ; Li et al. , 2019 ) stack filters of different sizes in a single layer to address the multiscale salient features . Dilated convolutions ( Pelt & Sethian , 2018 ; Wang et al. , 2018 ; Yu & Koltun , 2016 ; Yu et al. , 2017 ) , pyramid architectures ( Ke et al. , 2017 ; Lin et al. , 2017 ) , and multiscale dense networks ( Huang et al. , 2017 ) have also been proposed to take into account the multiscale feature information . Although the effectiveness of such models have been empirically demonstrated in various vision tasks , there is still a lack of interpretability of their ability to encode the input scale information . Group-equivariant CNNs , on the other hand , explicitly encode the group information into the network representation . Cohen & Welling ( 2016 ) proposed CNNs with group convolutions that are equivariant to several finite discrete symmetry groups . This idea is later generalized in Cohen et al . ( 2018 ) and applied mainly to the rotation groups SO ( 2 ) and SO ( 3 ) ( Cheng et al. , 2019 ; Weiler et al. , 2018a ; b ) . Although ST -equivariant CNNs have also been proposed in the literature ( Kanazawa et al. , 2014 ; Marcos et al. , 2018 ; Xu et al. , 2014 ; Ghosh & Gupta , 2019 ) , they are typically studied in a less general setting . In particular , none of these previous works proposed to conduct joint convolutions over R2 × S as a necessary and sufficient condition to impose equivariance , for which reason they are thus variants of a special case of our proposed architecture where the convolutional filters in S are Dirac delta functions ( c.f . Remark 1 . ) The scale-space semi-group correlation proposed in the concurrent work ( Worrall & Welling , 2019 ) bears the most resemblance to our proposed model , however , their approach is only limited to discrete semigroups , whereas our model does not have such restriction . Representation stability to input deformations . Input deformations typically induce noticeable variabilities within object classes , some of which are uninformative for the vision tasks . Models that are stable to input deformations are thus favorable in many applications . The scattering transform ( Bruna & Mallat , 2013 ; Mallat , 2010 ; 2012 ) computes translation-invariant representations that are Lipschitz continuous to deformations by cascading predefined wavelet transforms and modulus poolings . A joint convolution over R2 × SO ( 2 ) is later adopted in Sifre & Mallat ( 2013 ) to build roto-translation scattering with stable rotation/translation-invariant representations . These models , however , use pre-fixed wavelet transforms in the networks , and are thus nonadaptive to the data . DCFNet ( Qiu et al. , 2018 ) combines a pre-fixed filter basis and learnable expansion coefficients in a CNN architecture , achieving both data adaptivity and representation stability inherited from the filter regularity . This idea is later extended by Cheng et al . ( 2019 ) to produce SE ( 2 ) -equivariant representations that are Lipschitz continuous in L2 norm to input deformations modulo a global rotation , i.e. , the model stays approximately equivariant even if the input rotation is imperfect . To the best of our knowledge , a theoretical analysis of the deformation robustness of a ST -equivariant CNN has yet been studied , and a direct generalization of the result in Cheng et al . ( 2019 ) is futile because the feature maps of a ST -equivariant CNN is typically not in L2 ( c.f . Remark 2 . ) 3 ST -EQUIVARIANT CNN AND FILTER DECOMPOSITION . Group-equivariance is the property of a mapping f : X → Y to commute with the group actions on the domain X and codomain Y . More specifically , let G be a group , and Dg , Tg , respectively , be group actions on X and Y . A function f : X → Y is said to be G-equivariant if f ( Dgx ) = Tg ( f ( x ) ) , ∀ g ∈ G , x ∈ X . ( 1 ) G-invariance is thus a special case of G-equivariance where Tg = IdY . For learning tasks where the feature y ∈ Y is known a priori to change equivariantly to a group action g ∈ G on the input x ∈ X , e.g . image segmentation should be equivariant to translation , it would be beneficial to reduce the hypothesis space to include only G-equivaraint models . In this paper , we consider mainly the scaling-translation group ST ∼= S × R2 ∼= R × R2 . Given g = ( β , v ) ∈ ST and an input image x ( 0 ) ( u , λ ) ( u ∈ R2 is the spatial position , and λ is the unstructured channel index , e.g . RGB channels of a color image ) , the scaling-translation group action Dg = Dβ , v on x ( 0 ) is defined as Dβ , vx ( 0 ) ( u , λ ) : = x ( 0 ) ( 2−β ( u− v ) , λ ) . ( 2 ) Constructing ST -equivariant CNNs thus amounts to finding an architecture A such that each trained network f ∈ A commutes with the group action Dβ , v on the input and a similarly defined group action Tβ , v ( to be explained in Section 3.1 ) on the output . 3.1 ST -EQUIVARIANT CNNS . Inspired by Cheng et al . ( 2019 ) and Weiler et al . ( 2018b ) , we consider ST -equivariant CNNs with an extra index α ∈ S for the the scaling group S ∼= R : for each l ≥ 1 , the l-th layer output is denoted as x ( l ) ( u , α , λ ) , where u ∈ R2 is the spatial position , α ∈ S is the scale index , and λ ∈ [ Ml ] : = { 1 , . . . , Ml } corresponds to the unstructured channels . We use the continuous model for formal derivation , i.e. , the images and feature maps have continuous spatial and scale indices . In practice , the images are discretized on a Cartesian grid , and the scales are computed only on a discretized finite interval . Similar to Cheng et al . ( 2019 ) , the group action Tβ , v on the l-th layer output is defined as a scaling-translation in space as well as a shift in the scale channel : Tβ , vx ( l ) ( u , α , λ ) : = x ( l ) ( 2−β ( u− v ) , α− β , λ ) , ∀ l ≥ 1 . ( 3 ) A feedforward neural network is said to be scaling-translation-equivariant , i.e. , equivariant to ST , if x ( l ) [ Dβ , vx ( 0 ) ] = Tβ , vx ( l ) [ x ( 0 ) ] , ∀ l ≥ 1 , ( 4 ) where we slightly abuse the notation x ( l ) [ x ( 0 ) ] to denote the l-th layer output given the input x ( 0 ) . The following Theorem shows that ST -equivariance is achieved if and only if joint convolutions are conducted over S × R2 as in ( 5 ) and ( 6 ) . Theorem 1 . A feedforward neural network with an extra index α ∈ S for layerwise output is ST -equivariant if and only if the layerwise operations are defined as ( 5 ) and ( 6 ) : x ( 1 ) [ x ( 0 ) ] ( u , α , λ ) = σ ( ∑ λ′ ∫ R2 2−2αx ( 0 ) ( u+ u′ , λ′ ) W ( 1 ) λ′ , λ ( 2−αu′ ) du′ + b ( 1 ) ( λ ) ) , ( 5 ) x ( l ) [ x ( l−1 ) ] ( u , α , λ ) = σ ( ∑ λ′ ∫ R2 ∫ R 2−2αx ( l−1 ) ( u+ u′ , α+ α′ , λ′ ) · W ( l ) λ′ , λ ( 2−αu′ , α′ ) dα′du′ + b ( l ) ( λ ) ) , ∀l > 1 , ( 6 ) where σ : R→ R is a pointwise nonlinear function . We defer the proof of Theorem 1 , as well as those of other theorems , to the appendix . We note that the joint-convolution in Theorem 1 is a generalization of the group convolution proposed by Cohen & Welling ( 2016 ) to a non-compact group ST in the continuous setting . Remark 1 . When the ( joint ) convolutional filter W ( l ) λ′ , λ ( u , α ) takes the special form W ( l ) λ′ , λ ( u , α ) = V ( l ) λ′ , λ ( u ) δ ( α ) , the joint convolution ( 6 ) over R2×S reduces to only a ( multiscale ) spatial convolution x ( l ) [ x ( l−1 ) ] ( u , α , λ ) = σ ( ∑ λ′ ∫ R2 x ( l−1 ) ( u+ u′ , α , λ′ ) V ( l ) λ′ , λ ( 2−αu′ ) 2−2αdu′ + b ( l ) ( λ ) ) , ( 7 ) i.e. , the feature maps at different scales do not transfer information among each other ( see Figure 1a ) . The previous works ( Kanazawa et al. , 2014 ; Marcos et al. , 2018 ; Xu et al. , 2014 ; Ghosh & Gupta , 2019 ) on ST -equivariant CNNs are all based on this special case of Theorem 1 . Although the joint convolutions ( 6 ) on R2 × S provide the most general way of imposing ST - equivariance , they unfortunately also incur a significant increase in the model size and computational burden . Following the idea of Cheng et al . ( 2019 ) and Qiu et al . ( 2018 ) , we address this issue by taking a truncated decomposition of the convolutional filters under a pre-fixed separable basis , which will be discussed in detail in the next section .

Summary: The authors propose a CNN architecture, that is theoretically equivariant to isotropic scalings and translations. For this they add an extra scale-dimension to activation tensors, along with the existing two spatial dimensions. In practice they implement this with scale-steerable filters, which are discretised and truncated in both the spatial and scale-dimensions. They also provide a deformation robustness analysis.