Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation

1 INTRODUCTION . The rich diversity of object appearance in images arises from variations in object semantics and deformation . Semantics describe the high-level abstraction of what we perceive , and deformation defines the geometric transformation tied to specific data ( Gibson , 1950 ) . Humans are remarkably adept at making abstractions of the world ( Hudson & Manning , 2019 ) ; we see in raw visual signals , abstract semantics away from deformation , and form concepts . Interestingly , modern convolutional networks follow an analogous process by making abstractions through local connectivity and weight sharing ( Zhang , 2019 ) . However , such a mechanism is an inefficient one , as the emergent representations encode semantics and deformation together , instead of as disjoint notions . Though a convolution responds accordingly to each input , how it responds is primarily programmed by its rigid kernels , as in Figure 1 ( a , b ) . In effect , this consumes large model capacity and data modes ( Shelhamer et al. , 2019 ) . We argue that the awareness of deformations emerges from adaptivity – the ability to adapt at runtime ( Kanazawa et al. , 2016 ; Jia et al. , 2016 ; Li et al. , 2019 ) . Modeling of geometric transformations has been a constant pursuit for vision researchers over decades ( Lowe et al. , 1999 ; Lazebnik et al. , 2006 ; Jaderberg et al. , 2015 ; Dai et al. , 2017 ) . A basic idea is to spatially recompose data towards a common mode such that semantic recognition suffers less from deformation . A recent work that ∗Equal contributions . Work is done when Hang and Xizhou are interns at Microsoft Research Asia . is representative of this direction is Deformable Convolution ( Dai et al. , 2017 ; Zhu et al. , 2019 ) . As shown in Figure 1 ( c ) , it augments the convolutions with free-form sampling grids in the data space . It is previously justified as adapting receptive field , or what we phrase as the “ theoretical receptive field ” , that defines which input pixels can contribute to the final output . However , theoretical receptive field does not measure how much impact an input pixel actually has . On the other hand , Luo et al . ( 2016 ) propose to measure the effective receptive field ( ERF ) , i.e . the partial derivative of the output with respect to the input data , to quantify the exact contribution of each raw pixel to the convolution . Since adapting the theoretical receptive field is not the goal but a means to adapt the ERF , why not directly tune the ERF to specific data and tasks at runtime ? Toward this end , we introduce Deformable Kernels ( DKs ) , a family of novel and generic convolutional operators for deformation modeling . We aim to augment rigid kernels with the expressiveness to directly interact with the ERF of the computation during inference . Illustrated in Figure 1 ( d ) , DKs learn free-form offsets on kernel coordinates to deform the original kernel space towards specific data modality , rather than recomposing data . This can directly adapt ERF while leaving receptive field untouched . The design of DKs that is agnostic to data coordinates naturally leads to two variants – the global DK and the local DK , which behave differently in practice as we later investigate . We justify our approach with theoretical results which show that ERF is strictly determined by data sampling locations and kernel values . Used as a generic drop-in replacement of rigid kernels , DKs achieve empirical results coherent with our developed theory . Concretely , we evaluate our operator with standard base models on image classification and object detection . DKs perform favorably against prior works that adapt during runtime . With both quantitative and qualitative analysis , we further show that DKs can work orthogonally and complementarily with previous techniques . 2 RELATED WORKS . We distinguish our work within the context of deformation modeling as our goal , and dynamic inference as our means . Deformation Modeling : We refer to deformation modeling as learning geometric transformations in 2D image space without regard to 3D . One angle to attack deformation modeling is to craft certain geometric invariances into networks . However , this usually requires designs specific to certain kinds of deformation , such as shift , rotation , reflection and scaling ( Sifre & Mallat , 2013 ; Bruna & Mallat , 2013 ; Kanazawa et al. , 2016 ; Cohen & Welling , 2016 ; Worrall et al. , 2017 ; Esteves et al. , 2018 ) . Another line of work on this topic learns to recompose data by either semi-parameterized or completely free-form sampling in image space : Spatial Transformers ( Jaderberg et al. , 2015 ) learns 2D affine transformations , Deep Geometric Matchers ( Rocco et al. , 2017 ) learns thin-plate spline transformations , Deformable Convolutions ( Dai et al. , 2017 ; Zhu et al. , 2019 ) learns free-form transformations . We interpret sampling data space as an effective approach to adapt effective receptive fields ( ERF ) by directly changing receptive field . At a high-level , our Deformable Kernels ( DKs ) share intuitions with this line of works for learning geometric transformations , yet are instantiated by learning to sample in kernel space which directly adapt ERF while leaving theoretical receptive fields untouched . While kernel space sampling is also studied in Deformable Filter ( Xiong et al. , 2019 ) and KPConv ( Thomas et al. , 2019 ) , but in their contexts , sampling grids are computed from input point clouds rather than learned from data corpora . Dynamic Inference : Dynamic inference adapts the model or individual operators to the observed data . The computation of our approach differs from self-attention ( Vaswani et al. , 2017 ; Wang et al. , 2018 ) in which linear or convolution modules are augmented with subsequent queries that extract from the same input . We consider our closest related works in terms of implementation as those approaches that adapt convolutional kernels at run time . It includes but is not limited to Dynamic Filters ( Jia et al. , 2016 ) , Selective Kernels ( Li et al. , 2019 ) and Conditional Convolutions ( Yang et al. , 2019 ) . All of these approaches can learn and infer customized kernel spaces with respect to the data , but are either less inefficient or are loosely formulated . Dynamic Filters generate new filters from scratch , while Conditional Convolutions extend this idea to linear combinations of a set of synthesized filters . Selective Kernels are , on the other hand , comparably lightweight , but aggregating activations from kernels of different size is not as compact as directly sampling the original kernel space . Another line of works contemporary to ours ( Shelhamer et al. , 2019 ; Wang et al. , 2019 ) is to compose free-form filters with structured Gaussian filters , which essentially transforms kernel spaces by data . Our DKs also differ from these works with the emphasize of direct adaptation the ERF rather than the theoretical receptive field . As mentioned previously , the true goal should be to adapt the ERF , and to our knowledge , our work is the first to study dynamic inference of ERFs . 3 APPROACH . We start by covering preliminaries on convolutions , including the definition of effective receptive field ( ERF ) . We then formulate a theoretical framework for analyzing ERFs , and thus motivate our Deformable Kernels ( DKs ) . We finally elaborate different DK variants within such a framework . Our analysis suggests compatibility between DKs and the prior work . 3.1 A DIVE INTO CONVOLUTIONS . 2D Convolution : Let us first consider an input image I ∈ RD×D . By convolving it with a kernel W ∈ RK×K of stride 1 , we have an output image O whose pixel values at each coordinate j ∈ R2 can be expressed as Oj = ∑ k∈K Ij+kWk , ( 1 ) by enumerating discrete kernel positions k within the support K = [ −K/2 , K/2 ] 2∩Z . This defines a rigid grid for sampling data and kernels . Theoretical Receptive Field : The same kernel W can be stacked repeatedly to form a linear convolutional network with n layers . The theoretical receptive field can then be imagined as the “ accumulative coverage ” of kernels at each given output unit on the input image by deconvolving back through the network . This property characterizes a set of input fields that could fire percepts onto corresponding output pixels . The size of a theoretical receptive field scales linearly with respect to the network depth n and kernel size K ( He et al. , 2016 ) . Effective Receptive Field : Intuitively , not all pixels within a theoretical receptive field contribute equally . The influence of different fields varies from region to region thanks to the central emphasis of stacked convolutions and also to the non-linearity induced by activations . The notion of effective receptive field ( ERF ) ( Luo et al. , 2016 ) is thus introduced to measure the impact of each input pixel on the output at given locations . It is defined as a partial derivative field of the output with respect to the input data . With the numerical approximations in linear convolution networks , the ERF was previously identified as a Gaussian-like soft attention map over input images whose size grows fractionally with respect to the network depth n and linearly to the kernel size K. Empirical results validate this idea under more complex and realistic cases when networks exploit non-linearities , striding , padding , skip connections , and subsampling . 3.2 ANALYSIS ON EFFECTIVE RECEPTIVE FIELDS . We aim to revisit and complement the previous analysis on ERFs by Luo et al . ( 2016 ) . While the previous analysis concentrates on studying the expectation of an ERF , i.e. , when network depth n approaches infinity or all kernels are randomly distributed without learning in general , our analysis focuses on how we can perturb the computation such that the change in ERF is predictable , given an input and a set of kernel spaces . We start our analysis by considering a linear convolutional network , without any unit activations , as defined in Section 3.1 . For consistency , superscripts are introduced to image I , kernel W , and subscripts to kernel positions k to denote the index s ∈ [ 1 , n ] of each layer . Formally , given an input image I ( 0 ) and a set of K ×K kernels { W ( s ) } ns=1 of stride 1 , we can roll out the final output O ≡ I ( n ) by unfolding Equation 1 as I ( n ) j = ∑ kn∈K I ( n−1 ) j+kn W ( n ) kn = ∑ ( kn−1 , kn ) ∈K2 I ( n−2 ) j+kn+kn−1 W ( n ) kn W ( n−1 ) kn−1 = · · · = ∑ ( k1 , k2 , ... , kn ) ∈Kn ( I ( 0 ) j+ ∑n s=1 ks · n∏ s=1 W ( s ) ks ) . ( 2 ) By definition1 , the effective receptive field value R ( n ) ( i ; j ) ≡ ∂I ( n ) j /∂I ( 0 ) i of output coordinate j that takes input coordinate i can be computed by R ( n ) ( i ; j ) = ∑ ( k1 , k2 , ... , kn ) ∈Kn ( 1 [ j + n∑ s=1 ks = i ] · n∏ s=1 W ( s ) ks ) , ( 3 ) where 1 [ · ] denotes the indicator function . This result indicates that ERF is related only to the data sampling location j , kernel sampling location k , and kernel matrices { W ( s ) } . If we replace the mth kernel W ( m ) with a 1 × 1 kernel of a single parameter W ( m ) k̃m sampled from it , the value of ERF becomes to R ( n ) ( i ; j , k̃m ) = ∑ ( k1 , ... , km−1 , km+1 , ... , kn ) ∈Kn−1 ( 1 [ j + ∑ s∈S ks = i ] · ∏ s∈S W ( s ) ks ·W ( m ) k̃m ) , ( 4 ) where S = [ 1 , n ] \ { m } . Since a K×K kernel can be deemed as a composition of K2 1×1 kernels distributed on a square grid , Equation 3 can thus be reformulated as R ( n ) ( i ; j ) = ∑ km∈K R ( n ) ( i ; j + km , km ) . ( 5 ) For the case of complex non-linearities , where we here consider post ReLU2 activations in Equation 1 , Oj = max ( ∑ k∈K Ij+kWk , 0 ) . ( 6 ) 1 The original definition of ERF in Luo et al . ( 2016 ) focuses on the central coordinate of the output , i.e . j = ( 0 , 0 ) , to partially avoid the effects of zero padding . In this work , we will keep j in favor of generality while explicitly assuming input size D →∞ . 2 Our analysis currently only considers the ReLU network for its nice properties and prevalent popularity . We can follow a similar analysis and derive corresponding ERF as R ′ ( n ) ( i ; j , k̃m ) = ∑ ( k1 , ··· , km−1 , km+1 , ··· , kn ) ∈Kn−1 ( C ( n ) ( i ; j , k1 , · · · , kn , k̃m ) · ∏ s∈S W ( s ) ks ·Wmk̃m ) where C ( n ) ( i ; j , k1 , · · · , kn , k̃m ) = 1 [ j + ∑ s∈S ks = i ] ∏ s∈S 1 [ I ( s−1 ) j W ( s ) k̃s > 0 ] 1 [ I ( m−1 ) j W ( m ) k̃m > 0 ] . Here we can see that the ERF becomes data-dependent due to the coefficient C , which is tied to input coordinates , kernel sampling locations , and input data I ( 0 ) . The more detailed analysis of this coefficient is beyond the scope of this paper . However , it should be noted that this coefficient only “ gates ” the contribution of the input pixels to the output . So in practice , ERF is “ porous ” – there are inactive ( or gated ) pixel units irregularly distributed around the ones that fire . This phenomenon also appeared in previous studies ( such as in Luo et al . ( 2016 ) , Figure 1 ) . The maximal size of an ERF is still controlled by the data sampling location and kernel values as in the linear cases in Equation 5 . A nice property of Equation 4 and Equation 5 is that all computations are linear , making it compatible with any linear sampling operators for querying kernel values of fractional coordinates . In other words , sampling kernels in effect samples the ERF on the data in the linear case , but also roughly generalizes to non-linear cases as well . This finding motivates our design of Deformable Kernels ( DKs ) in Section 3.3 .

Summary: This paper introduce a simple algorithm called deformable kernels. It learns to generate a collection of coordinate offset Δk for each of the convolutional kernel element. Then during convolution, the kernel is treated as a 2D regular grid and sampled (interpolated) according to the generated coordinate offset before applying to the inputs. An auxiliary shallow network is learned to generate those coordinate offsets based in inputs. This method is very similar to the existing "deformable convolution" algorithm, though this operate on the kernels instead. Numerical experiments on image classification and object detection tasks show that the method performs better or comparably to strong baselines. It boost the performance even more when combined with existing methods.