Scale-Equivariant Steerable Networks

1 INTRODUCTION . Scale transformations occur in many image and video analysis tasks . They are a natural consequence of the variable distances among objects , or between objects and the camera . Such transformations result in significant changes in the input space which are often difficult for models to handle appropriately without careful consideration . At a high level , there are two modeling paradigms which allow a model to deal with scale changes : models can be endowed with an internal notion of scale and transform their predictions accordingly , or instead , models can be designed to be specifically invariant to scale changes . In image classification , when scale changes are commonly a factor of 2 , it is often sufficient to make class prediction independent of scale . However , in tasks such as image segmentation , visual tracking , or object detection , scale changes can reach factors of 10 or more . In these cases , it is intuitive that the ideal prediction should scale proportionally to the input . For example , the segmentation map of a nearby pedestrian should be easily converted to that of a distant person simply by downscaling . Convolutional Neural Networks ( CNNs ) demonstrate state-of-the-art performance in a wide range of tasks . Yet , despite their built-in translation equivariance , they do not have a particular mechanism for dealing with scale changes . One way to make CNNs account for scale is to train them with data augmentation Barnard & Casasent ( 1991 ) . This is , however , suitable only for global transformations . As an alternative , Henriques & Vedaldi ( 2017 ) and Tai et al . ( 2019 ) use the canonical coordinates of scale transformations to reduce scaling to well-studied translations . While these approaches do allow for scale equivariance , they consequently break translation equivariance . Several attempts have thus been made to extend CNNs to both scale and translation symmetry simultaneously . Some works use input or filter resizing to account for scaling in deep layers Xu et al . ( 2014 ) ; Kanazawa et al . ( 2014 ) . Such methods are suboptimal due to the time complexity of tensor resizing and the need for interpolation . In Ghosh & Gupta ( 2019 ) the authors pre-calculate filters defined on several scales to build scale-invariant networks , while ignoring the important case of scale equivariance . In contrast , Worrall & Welling ( 2019 ) employ the theory of semigroup equivariant networks with scale-space as an example ; however , this method is only suitable for integer downscale factors and therefore limited . ∗Correspondence to i.sosnovik @ uva.nl †Work was done when the author had an internship at the UvA-Bosch Delta Lab Source code is available at http : //github.com/isosnovik/sesn In this paper we develop a theory of scale-equivariant networks . We demonstrate the concept of steerable filter parametrization which allows for scaling without the need for tensor resizing . Then we derive scale-equivariant convolution and demonstrate a fast algorithm for its implementation . Furthermore , we experiment to determine to what degree the mathematical properties actually hold true . Finally , we conduct a set of experiments comparing our model with other methods for scale equivariance and local scale invariance . The proposed model has the following advantages compared to other scale-equivariant models : 1 . It is equivariant to scale transformations with arbitrary discrete scale factors and is not limited to either integer scales or scales tailored by the image pixel grid . 2 . It does not rely on any image resampling techniques during training , and therefore , produces deep scale-equivariant representations free of any interpolation artifacts . 3 . The algorithm is based on the combination of tensor expansion and 2-dimensional convolution , and demonstrates the same computation time as the general CNN with a comparable filter bank . 2 PRELIMINARIES . Before we move into scale-equivariant mappings , we discuss some aspects of equivariance , scaling transformations , symmetry groups , and the functions defined on them . For simplicity , in this section , we consider only 1-dimensional functions . The generalization to higher-dimensional cases is straightforward . Equivariance Let us consider some mapping g. It is equivariant under Lθ if and only if there exists L′θ such that g ◦ Lθ = L′θ ◦ g. In case L′θ is the identity mapping , the function g is invariant . In this paper we consider scaling transformations . In order to guarantee the equivariance of the predictions to such transformations , and to improve the performance of the model , we seek to incorporate this property directly inside CNNs . Scaling Given a function f : R→ R , a scale transformation is defined as follows : Ls [ f ] ( x ) = f ( s −1x ) , ∀s > 0 ( 1 ) We refer to cases with s > 1 as upscale and to cases with s < 1 as downscale . If we convolve the downscaled function with an arbitrary filter ψ and perform a simple change of variables inside the integral , we get the following property : [ Ls [ f ] ? ψ ] ( x ) = ∫ R Ls [ f ] ( x ′ ) ψ ( x′ − x ) dx′ = ∫ R f ( s−1x′ ) ψ ( x′ − x ) dx′ = s ∫ R f ( s−1x′ ) ψ ( s ( s−1x′ − s−1x ) ) d ( s−1x′ ) = sLs [ f ? Ls−1 [ ψ ] ] ( x ) ( 2 ) In other words , convolution of the downscaled function with a filter can be expressed through a convolution of the function with the correspondingly upscaled filter where downscaling is performed afterwards . Equation 2 shows us that the standard convolution is not scale-equivariant . Steerable Filters In order to make computations simpler , we reparametrize ψσ ( x ) = σ−1ψ ( σ−1x ) , which has the following property : Ls−1 [ ψσ ] ( x ) = ψσ ( sx ) = s −1ψs−1σ ( x ) ( 3 ) It gives a shorter version of Equation 2 : Ls [ f ] ? ψσ = Ls [ f ? ψs−1σ ] ( 4 ) We will refer to such a parameterization of filters as Steerable Filters because the scaling of these filters is the transformation of its parameters . Note that we may construct steerable filters from any function . This has the important consequence that it does not restrict our approach . Rather it will make the analysis easier for discrete data . Moreover , note that any linear combination of steerable filters is still steerable . Scale-Translation Group All possible scales form the scaling group S. Here we consider the discrete scale group , i.e . scales of the form . . . a−1 , a−1 , 1 , a , a2 , . . . with base a as a parameter of our method . Analysis of this group by itself breaks the translation equivariance of CNNs . Thus we seek to incorporate scale and translation symmetries into CNNs , and , therefore consider the ScaleTranslation Group H . It is a semidirect product of the scaling group S and the group of translations T ∼= R. In other words : H = { ( s , t ) |s ∈ S , t ∈ T } . For multiplication of group elements , we have ( s2 , t2 ) · ( s1 , t1 ) = ( s2s1 , s2t1 + t2 ) and for the inverse ( s2 , t2 ) −1 · ( s1 , t1 ) = ( s−12 s1 , s −1 2 ( t1− t2 ) ) . Additionally , for the corresponding scaling and translation transformations , we have Lst = LsLt 6= LtLs , which means that the order of the operations matters . From now on , we will work with functions defined on groups , i.e . mappings H → R. Note , that simple function f : R → R may be considered as a function on H with constant value along the S axis . Therefore , Equation 4 holds true for functions on H as well . One thing we should keep in mind is that when we apply Ls to functions on H and R we use different notations . For example Ls [ f ] ( x ′ ) = f ( s−1x′ ) and Ls [ f ] ( s′ , t′ ) = f ( ( s , 0 ) −1 ( s′ , t′ ) ) = f ( s−1s′ , s−1t′ ) Group-Equivariant Convolution Given group G and two functions f and ψ defined on it , Gequivariant convolution is given by [ f ? G ψ ] ( g ) = ∫ G f ( g′ ) Lg [ ψ ] ( g ′ ) dµ ( g′ ) = ∫ G f ( g′ ) ψ ( g−1g′ ) dµ ( g′ ) ( 5 ) Here µ ( g′ ) is the Haar measure also known as invariant measure Folland ( 2016 ) . For T ∼= R we have dµ ( g′ ) = dg′ . For discrete groups , the Haar measure is the counting measure , and integration becomes a discrete sum . This formula tells us that the output of the convolution evaluated at point g is the inner product between the function f and the transformed filter Lg [ ψ ] . 3 SCALE-EQUIVARIANT MAPPINGS . Now we define the main building blocks of scale-equivariant models . Scale Convolution In order to derive scale convolution , we start from group equivariant convolution with G = H . We first use the property of semidirect product of groups which splits the integral , then choose the appropriate Haar measures , and finally use the properties of steerable filters . Given the function f ( s , t ) and a steerable filter ψσ ( s , t ) defined on H , a scale convolution is given by : [ f ? H ψσ ] ( s , t ) = ∫ S ∫ T f ( s′ , t′ ) Lst [ ψσ ] ( s ′ , t′ ) dµ ( s′ ) dµ ( t′ ) = ∑ s′ ∫ T f ( s′ , t′ ) ψsσ ( s −1s′ , t′ − t ) dt′ = ∑ s′ [ f ( s′ , · ) ? ψsσ ( s−1s′ , · ) ] ( t ) ( 6 ) And for the case of Cin input and Cout output channels we have : [ f ? H ψσ ] m ( s , t ) = Cin∑ n=1 ∑ s′ [ fn ( s ′ , · ) ? ψn , m , sσ ( s−1s′ , · ) ] ( t ) , m = 1 . . . Cout ( 7 ) The proof of the equivariance of this convolution to transformations fromH is given in Appendix A. Kondor & Trivedi ( 2018 ) prove that a feed-forward neural network is equivariant to transformations from G if and only if it is constructed from G-equivariant convolutional layers . Thus Equation 7 shows the most general form of scale-equivariant layers which allows for building scale-equivariant convolutional networks with such choice of S. We will refer to models using scale-equivariant layers with steerable filters as Scale-Equivariant Steerable Networks , or shortly SESN1 Nonlinearities In order to guarantee the equivariance of the network to scale transformations , we use scale equivariant nonlinearities . We are free to use simple point-wise nonlinearities . Indeed , 1pronounced ‘ season ’ point-wise nonlinearities ν , like ReLU , commute with scaling transformations : [ ν ◦ Ls [ f ] ] ( s′ , x′ ) = ν ( Ls [ f ] ( s′ , x′ ) ) = ν ( f ( s−1s′ , s−1x′ ) ) = ν [ f ] ( s−1s′ , s−1x′ ) = [ Ls ◦ ν [ f ] ] ( s′ , x′ ) ( 8 ) Pooling Until now we did not discuss how to convert an equivariant mapping to invariant one . One way to do this is to calculate the invariant measure of the signal . In case of translation , such a measure could be the maximum value for example . First , we propose the maximum scale projection defined as f ( s , x ) → maxs f ( s , x ) . This transformation projects the function f from H to T . Therefore , the representation stays equivariant to scaling , but loses all information about the scale itself . Second , we are free to use spatial max-pooling with a moving window or global max pooling . Transformation f ( s , x ) → maxx f ( s , x ) projects the function f fromH to S. The obtained representation is invariant to scaling in spatial domain , however , it stores the information about scale . Finally , we can combine both of these pooling mechanisms in any order . The obtained transformation produces a scale invariant function . It is useful to utilize this transformation closer to the end of the network , when the deep representation must be invariant to nuisance input variations , but already has very rich semantic meaning .

Summary: This paper proposes a framework (SESN) for learning deep networks that possess scale equivariance in addition to translation invariance. The formulation is based on group convolution on the scale-translation group. Filters are represented as the coefficients of a set of continuous basis functions, which are sampled (once) at a discrete set of scales. The theoretical formulatioin is clear and interesting. The approach is evaluated in terms of image classification accuracy. The set of baselines is quite exhaustive, including recent papers and papers that are not widely-known.