Scale-Equivariant Steerable Networks

1 INTRODUCTION . Scale transformations occur in many image and video analysis tasks . They are a natural consequence of the variable distances among objects , or between objects and the camera . Such transformations result in significant changes in the input space which are often difficult for models to handle appropriately without careful consideration . At a high level , there are two modeling paradigms which allow a model to deal with scale changes : models can be endowed with an internal notion of scale and transform their predictions accordingly , or instead , models can be designed to be specifically invariant to scale changes . In image classification , when scale changes are commonly a factor of 2 , it is often sufficient to make class prediction independent of scale . However , in tasks such as image segmentation , visual tracking , or object detection , scale changes can reach factors of 10 or more . In these cases , it is intuitive that the ideal prediction should scale proportionally to the input . For example , the segmentation map of a nearby pedestrian should be easily converted to that of a distant person simply by downscaling . Convolutional Neural Networks ( CNNs ) demonstrate state-of-the-art performance in a wide range of tasks . Yet , despite their built-in translation equivariance , they do not have a particular mechanism for dealing with scale changes . One way to make CNNs account for scale is to train them with data augmentation Barnard & Casasent ( 1991 ) . This is , however , suitable only for global transformations . As an alternative , Henriques & Vedaldi ( 2017 ) and Tai et al . ( 2019 ) use the canonical coordinates of scale transformations to reduce scaling to well-studied translations . While these approaches do allow for scale equivariance , they consequently break translation equivariance . Several attempts have thus been made to extend CNNs to both scale and translation symmetry simultaneously . Some works use input or filter resizing to account for scaling in deep layers Xu et al . ( 2014 ) ; Kanazawa et al . ( 2014 ) . Such methods are suboptimal due to the time complexity of tensor resizing and the need for interpolation . In Ghosh & Gupta ( 2019 ) the authors pre-calculate filters defined on several scales to build scale-invariant networks , while ignoring the important case of scale equivariance . In contrast , Worrall & Welling ( 2019 ) employ the theory of semigroup equivariant networks with scale-space as an example ; however , this method is only suitable for integer downscale factors and therefore limited . ∗Correspondence to i.sosnovik @ uva.nl †Work was done when the author had an internship at the UvA-Bosch Delta Lab Source code is available at http : //github.com/isosnovik/sesn In this paper we develop a theory of scale-equivariant networks . We demonstrate the concept of steerable filter parametrization which allows for scaling without the need for tensor resizing . Then we derive scale-equivariant convolution and demonstrate a fast algorithm for its implementation . Furthermore , we experiment to determine to what degree the mathematical properties actually hold true . Finally , we conduct a set of experiments comparing our model with other methods for scale equivariance and local scale invariance . The proposed model has the following advantages compared to other scale-equivariant models : 1 . It is equivariant to scale transformations with arbitrary discrete scale factors and is not limited to either integer scales or scales tailored by the image pixel grid . 2 . It does not rely on any image resampling techniques during training , and therefore , produces deep scale-equivariant representations free of any interpolation artifacts . 3 . The algorithm is based on the combination of tensor expansion and 2-dimensional convolution , and demonstrates the same computation time as the general CNN with a comparable filter bank . 2 PRELIMINARIES . Before we move into scale-equivariant mappings , we discuss some aspects of equivariance , scaling transformations , symmetry groups , and the functions defined on them . For simplicity , in this section , we consider only 1-dimensional functions . The generalization to higher-dimensional cases is straightforward . Equivariance Let us consider some mapping g. It is equivariant under Lθ if and only if there exists L′θ such that g ◦ Lθ = L′θ ◦ g. In case L′θ is the identity mapping , the function g is invariant . In this paper we consider scaling transformations . In order to guarantee the equivariance of the predictions to such transformations , and to improve the performance of the model , we seek to incorporate this property directly inside CNNs . Scaling Given a function f : R→ R , a scale transformation is defined as follows : Ls [ f ] ( x ) = f ( s −1x ) , ∀s > 0 ( 1 ) We refer to cases with s > 1 as upscale and to cases with s < 1 as downscale . If we convolve the downscaled function with an arbitrary filter ψ and perform a simple change of variables inside the integral , we get the following property : [ Ls [ f ] ? ψ ] ( x ) = ∫ R Ls [ f ] ( x ′ ) ψ ( x′ − x ) dx′ = ∫ R f ( s−1x′ ) ψ ( x′ − x ) dx′ = s ∫ R f ( s−1x′ ) ψ ( s ( s−1x′ − s−1x ) ) d ( s−1x′ ) = sLs [ f ? Ls−1 [ ψ ] ] ( x ) ( 2 ) In other words , convolution of the downscaled function with a filter can be expressed through a convolution of the function with the correspondingly upscaled filter where downscaling is performed afterwards . Equation 2 shows us that the standard convolution is not scale-equivariant . Steerable Filters In order to make computations simpler , we reparametrize ψσ ( x ) = σ−1ψ ( σ−1x ) , which has the following property : Ls−1 [ ψσ ] ( x ) = ψσ ( sx ) = s −1ψs−1σ ( x ) ( 3 ) It gives a shorter version of Equation 2 : Ls [ f ] ? ψσ = Ls [ f ? ψs−1σ ] ( 4 ) We will refer to such a parameterization of filters as Steerable Filters because the scaling of these filters is the transformation of its parameters . Note that we may construct steerable filters from any function . This has the important consequence that it does not restrict our approach . Rather it will make the analysis easier for discrete data . Moreover , note that any linear combination of steerable filters is still steerable . Scale-Translation Group All possible scales form the scaling group S. Here we consider the discrete scale group , i.e . scales of the form . . . a−1 , a−1 , 1 , a , a2 , . . . with base a as a parameter of our method . Analysis of this group by itself breaks the translation equivariance of CNNs . Thus we seek to incorporate scale and translation symmetries into CNNs , and , therefore consider the ScaleTranslation Group H . It is a semidirect product of the scaling group S and the group of translations T ∼= R. In other words : H = { ( s , t ) |s ∈ S , t ∈ T } . For multiplication of group elements , we have ( s2 , t2 ) · ( s1 , t1 ) = ( s2s1 , s2t1 + t2 ) and for the inverse ( s2 , t2 ) −1 · ( s1 , t1 ) = ( s−12 s1 , s −1 2 ( t1− t2 ) ) . Additionally , for the corresponding scaling and translation transformations , we have Lst = LsLt 6= LtLs , which means that the order of the operations matters . From now on , we will work with functions defined on groups , i.e . mappings H → R. Note , that simple function f : R → R may be considered as a function on H with constant value along the S axis . Therefore , Equation 4 holds true for functions on H as well . One thing we should keep in mind is that when we apply Ls to functions on H and R we use different notations . For example Ls [ f ] ( x ′ ) = f ( s−1x′ ) and Ls [ f ] ( s′ , t′ ) = f ( ( s , 0 ) −1 ( s′ , t′ ) ) = f ( s−1s′ , s−1t′ ) Group-Equivariant Convolution Given group G and two functions f and ψ defined on it , Gequivariant convolution is given by [ f ? G ψ ] ( g ) = ∫ G f ( g′ ) Lg [ ψ ] ( g ′ ) dµ ( g′ ) = ∫ G f ( g′ ) ψ ( g−1g′ ) dµ ( g′ ) ( 5 ) Here µ ( g′ ) is the Haar measure also known as invariant measure Folland ( 2016 ) . For T ∼= R we have dµ ( g′ ) = dg′ . For discrete groups , the Haar measure is the counting measure , and integration becomes a discrete sum . This formula tells us that the output of the convolution evaluated at point g is the inner product between the function f and the transformed filter Lg [ ψ ] . 3 SCALE-EQUIVARIANT MAPPINGS . Now we define the main building blocks of scale-equivariant models . Scale Convolution In order to derive scale convolution , we start from group equivariant convolution with G = H . We first use the property of semidirect product of groups which splits the integral , then choose the appropriate Haar measures , and finally use the properties of steerable filters . Given the function f ( s , t ) and a steerable filter ψσ ( s , t ) defined on H , a scale convolution is given by : [ f ? H ψσ ] ( s , t ) = ∫ S ∫ T f ( s′ , t′ ) Lst [ ψσ ] ( s ′ , t′ ) dµ ( s′ ) dµ ( t′ ) = ∑ s′ ∫ T f ( s′ , t′ ) ψsσ ( s −1s′ , t′ − t ) dt′ = ∑ s′ [ f ( s′ , · ) ? ψsσ ( s−1s′ , · ) ] ( t ) ( 6 ) And for the case of Cin input and Cout output channels we have : [ f ? H ψσ ] m ( s , t ) = Cin∑ n=1 ∑ s′ [ fn ( s ′ , · ) ? ψn , m , sσ ( s−1s′ , · ) ] ( t ) , m = 1 . . . Cout ( 7 ) The proof of the equivariance of this convolution to transformations fromH is given in Appendix A. Kondor & Trivedi ( 2018 ) prove that a feed-forward neural network is equivariant to transformations from G if and only if it is constructed from G-equivariant convolutional layers . Thus Equation 7 shows the most general form of scale-equivariant layers which allows for building scale-equivariant convolutional networks with such choice of S. We will refer to models using scale-equivariant layers with steerable filters as Scale-Equivariant Steerable Networks , or shortly SESN1 Nonlinearities In order to guarantee the equivariance of the network to scale transformations , we use scale equivariant nonlinearities . We are free to use simple point-wise nonlinearities . Indeed , 1pronounced ‘ season ’ point-wise nonlinearities ν , like ReLU , commute with scaling transformations : [ ν ◦ Ls [ f ] ] ( s′ , x′ ) = ν ( Ls [ f ] ( s′ , x′ ) ) = ν ( f ( s−1s′ , s−1x′ ) ) = ν [ f ] ( s−1s′ , s−1x′ ) = [ Ls ◦ ν [ f ] ] ( s′ , x′ ) ( 8 ) Pooling Until now we did not discuss how to convert an equivariant mapping to invariant one . One way to do this is to calculate the invariant measure of the signal . In case of translation , such a measure could be the maximum value for example . First , we propose the maximum scale projection defined as f ( s , x ) → maxs f ( s , x ) . This transformation projects the function f from H to T . Therefore , the representation stays equivariant to scaling , but loses all information about the scale itself . Second , we are free to use spatial max-pooling with a moving window or global max pooling . Transformation f ( s , x ) → maxx f ( s , x ) projects the function f fromH to S. The obtained representation is invariant to scaling in spatial domain , however , it stores the information about scale . Finally , we can combine both of these pooling mechanisms in any order . The obtained transformation produces a scale invariant function . It is useful to utilize this transformation closer to the end of the network , when the deep representation must be invariant to nuisance input variations , but already has very rich semantic meaning .

Summary: This paper proposed scale-equivariant steerable convolutional neural networks that is able to preserve both the translation and scaling symmetry of the data in the representation. To achieve this, the authors developed the scale-convolution blocks in the network, and generalized other common blocks, such as pooling and nonlinearity, to remain scale-equivariant. Extensive experiments have been conducted to show that the proposed scale-equivariant network