Blurring Structure and Learning to Optimize and Adapt Receptive Fields

The visual world is vast and varied , but its variations divide into structured and unstructured factors . We compose free-form filters and structured Gaussian filters , optimized end-to-end , to factorize deep representations and learn both local features and their degree of locality . In effect this optimizes over receptive field size and shape , tuning locality to the data and task . Our semi-structured composition is strictly more expressive than free-form filtering , and changes in its structured parameters would require changes in architecture for standard networks . Dynamic inference , in which the Gaussian structure varies with the input , adapts receptive field size to compensate for local scale variation . Optimizing receptive field size improves semantic segmentation accuracy on Cityscapes by 1-2 points for strong dilated and skip architectures and by up to 10 points for suboptimal designs . Adapting receptive fields by dynamic Gaussian structure further improves results , equaling the accuracy of free-form deformation while improving efficiency . 1 INTRODUCTION . Although the visual world is varied , it nevertheless has ubiquitous structure . Structured factors , such as scale , admit clear theories and efficient representation design . Unstructured factors , such as what makes a cat look like a cat , are too complicated to model analytically , requiring free-form representation learning . How can recognition harness structure without restraining the representation ? Free-form representations are structure-agnostic , making them general , but not exploiting structure is computationally and statistically inefficient . Structured representations like steerable filtering ( Freeman & Adelson , 1991 ; Simoncelli & Freeman , 1995 ; Jacobsen et al. , 2016 ) , scattering ( Bruna & Mallat , 2013 ; Sifre & Mallat , 2013 ) , and steerable networks ( Cohen & Welling , 2017 ) are efficient but constrained to the chosen structures . We propose a new , semi-structured compositional filtering approach to blur the line between free-form and structured representations and learn both . Doing so learns local features and the degree of locality . Free-form filters , directly defined by the parameters , are general and able to cope with unknown variations , but are parameter inefficient . Structured factors , such as scale and orientation , are enumerated like any other variation , and require duplicated learning across different layers and channels . Nonetheless , end-to-end learning of free-form parameters is commonly the most accurate approach to complex visual recognition tasks when there is sufficient data . Structured filters , indirectly defined as a function of the parameters , are theoretically clear and parameter efficient , but constrained . Their effectiveness hinges on whether or not they encompass the true structure of the data . If not , the representation is limiting , and subject to error . At least , this is a danger when substituting structure to replace learning . We compose free-form and structured filters , as shown in Figure 1 , and learn both end-to-end . Freeform filters are not constrained by our composition . This makes our approach more expressive , not less , while still able to efficiently learn the chosen structured factors . In this way our semi-structured networks can reduce to existing networks as a special case . At the same time , our composition can learn different receptive fields that can not be realized in the standard parameterization of free-form filters . Adding more free-form parameters or dilating can not learn the same family of filters . Figure 2 offers one example of the impracticality of architectural alternatives . 7 × 7 ( a ) composition ( b ) dilation ( c ) large kernel 7 × 7 + 3 7 × 7 21 × 21 f✓ < latexit sha1_base64= '' eXyhoyux/JZPwZbafyhHY5Q+VPA= '' > AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeCCB4r2A9oQ9lsN+3SzSbuToQS+ie8eFDEq3/Hm//GbZuDtj4YeLw3w8y8IJHCoOt+O4W19Y3NreJ2aWd3b/+gfHjUMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2Mb2Z++4lrI2L1gJOE+xEdKhEKRtFKnbDfwxFH2i9X3Ko7B1klXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJp6VeanhC2ZgOeddSRSNu/Gx+75ScWWVAwljbUkjm6u+JjEbGTKLAdkYUR2bZm4n/ed0Uw2s/EypJkSu2WBSmkmBMZs+TgdCcoZxYQpkW9lbCRlRThjaikg3BW355lbRqVe+iWru/rNRv8ziKcAKncA4eXEEd7qABTWAg4Rle4c15dF6cd+dj0Vpw8plj+APn8wcfwJAI < /latexit > g⌃ < latexit sha1_base64= '' AGrXzx9ht3BHt0C1BcEaSi4vZL0= '' > AAAB73icbVBNSwMxEJ2tX7V+VT16CRbBU9mtBT0WRPBY0X5Au5Rsmt2GJtk1yQpl6Z/w4kERr/4db/4b03YP2vpg4PHeDDPzgoQzbVz32ymsrW9sbhW3Szu7e/sH5cOjto5TRWiLxDxW3QBrypmkLcMMp91EUSwCTjvB+Hrmd56o0iyWD2aSUF/gSLKQEWys1I0G/XsWCTwoV9yqOwdaJV5OKpCjOSh/9YcxSQWVhnCsdc9zE+NnWBlGOJ2W+qmmCSZjHNGepRILqv1sfu8UnVlliMJY2ZIGzdXfExkWWk9EYDsFNiO97M3E/7xeasIrP2MySQ2VZLEoTDkyMZo9j4ZMUWL4xBJMFLO3IjLCChNjIyrZELzll1dJu1b1Lqq1u3qlcZPHUYQTOIVz8OASGnALTWgBAQ7P8ApvzqPz4rw7H4vWgpPPHMMfOJ8/6KSP5A== < /latexit > Figure 2 : Our composition does not reduce to dilation or more free-form parameters . Dilation is sparse , causing artifacts , and can not be learned . More free-form parameters require more data to learn and the maximum size is still bounded . Our contributions include : ( 1 ) composing Gaussian filtering and free-form filtering to bridge classic ideas for scale-space representation design and current practices for representation learning , ( 2 ) exploring a variety of receptive fields that our approach can learn , and ( 3 ) adapting receptive fields during inference with accurate and efficient dynamic Gaussian structure . 2 RELATED WORK . Composing structured Gaussian filters with free-form learned filters draws on structured filter design and representation learning . Our work is inspired by the transformation invariance of scale-space ( Lindeberg , 1994 ) , the parsimony of steerable filtering ( Freeman & Adelson , 1991 ; Perona , 1995 ; Bruna & Mallat , 2013 ; Cohen & Welling , 2017 ) , and the adaptivity of dynamic inference ( Olshausen et al. , 1993 ; Jaderberg et al. , 2015 ; De Brabandere et al. , 2016 ; Dai et al. , 2017 ) . Analysis that the effective receptive field size of deep networks is limited ( Luo et al. , 2016 ) , and is only a fraction of the theoretical size , motivates our goal of learning and adapting unbounded receptive field sizes and varied receptive field shapes . Transformation Invariance Gaussian scale-space and its affine extension connect covariance to spatial structure for transformation invariance ( Lindeberg , 1994 ) . We jointly learn structured transformations via Gaussian covariance and features via free-form filtering . Enumerative methods cover a set of transformations , rather than learning to select transformations : image pyramids ( Burt & Adelson , 1983 ) and feature pyramids ( Kanazawa et al. , 2014 ; Shelhamer et al. , 2017 ; Lin et al. , 2017 ) cover scale , scattering ( Bruna & Mallat , 2013 ) covers scales and rotations , and steerable networks ( Cohen & Welling , 2017 ) cover discrete groups . Our learning and inferring covariance relates to scale selection ( Lindeberg , 1998 ) , as exemplified by the scale invariant feature transform ( Lowe , 2004 ) . Scale-adaptive convolution ( Zhang et al. , 2017 ) likewise selects scales , but without our Gaussian structure and smoothness . Steering Steering indexes a continuous family of filters by linearly weighting a structured basis , such as Gaussian derivatives . Steerable filters ( Freeman & Adelson , 1991 ) index orientation and deformable kernels ( Perona , 1995 ) index orientation and scale . Such filters can be stacked into a deep , structured network ( Jacobsen et al. , 2016 ) . These methods have elegant structure , but are constrained to it . We make use of Gaussian structure , but keep generality by composing with free-form filters . Dynamic Inference Dynamic inference adapts the model to each input . Dynamic routing ( Olshausen et al. , 1993 ) , spatial transformers ( Jaderberg et al. , 2015 ) , dynamic filter nets ( De Brabandere et al. , 2016 ) , and deformable convolution ( Dai et al. , 2017 ) are all dynamic , but lack local structure . We incorporate Gaussian structure to improve efficiency while preserving accuracy . Proper signal processing , by blurring when downsampling , improves the shift-equivariance of learned filtering ( Zhang , 2019 ) . We reinforce these results with our experiments on blurred dilation , to complement their focus on blurred stride . While we likewise blur , and confirm the need for smoothing to prevent aliasing , our focus is on how to jointly learn and compose structured and free-form filters . 3 A CLEAR REVIEW OF BLURRING . We introduce our chosen structured filters first , and then compose them with free-form filters in the next section . While the Gaussian and scale-space ideas here are classic , our end-to-end optimized composition and its use for receptive field learning are novel . 3.1 GAUSSIAN STRUCTURE . The choice of Gaussian structure determines the filter characteristics that can be represented and learned . For learning , it is differentiable , low-dimensional for parameter efficiency , and expressive enough for different sizes and shapes . For signal processing , it is smooth and computationally efficient . In particular , the Gaussian has these attractive properties for our purposes : • shift-invariance for convolutional filtering , • normalization to preserve input and gradient norms for stable optimization , • separability to reduce computation by replacing a 2D filter with two 1D filters , • and cascade smoothing from semi-group structure to decompose filtering into smaller , cumulative steps . In fact , the Gaussian is the unique filter satisfying these and further scale-space axioms ( Koenderink , 1984 ; Babaud et al. , 1986 ; Lindeberg , 1994 ) . The Gaussian kernel in 2D is G ( x ; Σ ) = 1 2π √ det Σ e−x T Σ−1x/2 for input coordinates x and covariance Σ ∈ R2×2 , a symmetric positive-definite matrix . The structure of the Gaussian is controlled by its covariance Σ . Note that we are concerned with the spatial covariance , where the coordinates are considered as random variables , and not the covariance of the feature dimensions . Therefore the elements of the covariance matrix are σ2y , σ 2 x for the y , x coordinates and ρ for their correlation . The standard , isotropic Gaussian has identity covariance [ 1 00 1 ] . There is progressively richer structure in spherical [ σ2 0 0 σ2 ] , diagonal [ σ2y 0 0 σ2x ] , and full [ σ2y ρ ρ σ2x ] covariances which can represent scale , aspect , and orientation in one , two , or three parameters . Selecting the right covariance yields invariance to a given spatial transformation ( Lindeberg , 1994 ) . We leverage this transformation property of Gaussians to learn receptive fields in Section 4.1 and dynamically adapt them for locally invariant filtering in Section 4.2 . From the Gaussian kernel G ( x , Σ ) we instantiate a Gaussian filter gΣ ( · ) in the standard way : ( 1 ) evaluate the kernel at the coordinates of the filter coefficients and ( 2 ) renormalize by the sum to correct for this discretization . We decide the filter size according to the covariance by setting the half size = d2σe in each dimension . This covers ± 2σ to include 95 % of the true density no matter the covariance . ( We found that higher coverage did not improve our results . ) Our filters are always odd-sized to keep coordinates centered . 3.2 COVARIANCE PARAMETERIZATION & OPTIMIZATION . The covariance Σ is symmetric positive definite , requiring proper parameterization for unconstrained optimization . We choose the log-Cholesky parameterization ( Pinheiro & Bates , 1996 ) for iterative optimization because it is simple and quick to compute : Σ = U ′U for upper-triangularU with positive diagonal . We keep the diagonal positive by storing its log , hence log-Cholesky , and exponentiating when forming Σ . ( See Pinheiro & Bates ( 1996 ) for a primer on covariance parameterization . ) Here is an example for full covariance Σ with σ2y , σ 2 x for the y , x coordinates and ρ for their correlation : [ σ2y ρ ρ σ2x ] ← [ +1 −2 −2 +8 ] = [ +1 +0 −2 +2 ] [ +1 −2 +0 +2 ] = ( log ( 1 ) , −2 , log ( 2 ) ) . Spherical and diagonal covariance are parameterized by fixing ρ = 0 and tying/untying σy , σx . 3.3 LEARNING TO BLUR . As a pedagogical example , consider the problem of optimizing covariance to reproduce an unknown blur . That is , given a reference image and a blurred version of it , which Gaussian filter causes this blur ? Figure 3 shows such an optimization : from an identity-like initialization the covariance parameters quickly converge to the true Gaussian . Given the full covariance parameterization , optimization controls scale , aspect , and orientation . Each degree of freedom can be seen across the iterates of this example . Had the true blur been simpler , for instance spherical , it could still be swiftly recovered in the full parameterization . Notice how the size and shape of the filter vary over the course of optimization : this is only possible through structure . For a Gaussian filter , its covariance is the intrinsic structure , and its coefficients follow from it . The filter size and shape change while the dimension of the covariance itself is constant . Lacking structure , free-form parameterization couples the number of parameters and filter size , and so can not search over size and shape in this fashion .

Summary: This paper proposes semi-structured neural filter composed of structured Gaussian filters and the usual structure-agnostic free-form filters found in neural networks. They are optimized using end-to-end training. Effectively, this lead to increased receptive field size and shape with just few additional parameters. Further, this module is architecture agnostic and can also be integrated with any dynamic inference models. Specifically, when applied on deformable convolutional filters, the deformation at each input can be structured using gaussian filters. Empirical experiments suggest that when integrated with state-of-the-art semantic segmentation architectures, the absolute accuracy on Cityscapes improves by 2%. Large improvement in seen on naive / sub-optimal architectures for segmentation.