Title: BLURRING STRUCTURE AND LEARNING TO OPTIMIZE AND ADAPT RECEPTIVE FIELDS

Abstract: The visual world is vast and varied, but its variations divide into structured and unstructured factors. We compose free-form filters and structured Gaussian filters, optimized end-to-end, to factorize deep representations and learn both local features and their degree of locality. In effect this optimizes over receptive field size and shape, tuning locality to the data and task. Our semi-structured composition is strictly more expressive than free-form filtering, and changes in its structured parameters would require changes in architecture for standard networks. Dynamic inference, in which the Gaussian structure varies with the input, adapts receptive field size to compensate for local scale variation. Optimizing receptive field size improves semantic segmentation accuracy on Cityscapes by 1-2 points for strong dilated and skip architectures and by up to 10 points for suboptimal designs. Adapting receptive fields by dynamic Gaussian structure further improves results, equaling the accuracy of free-form deformation while improving efficiency.

1. INTRODUCTION:

Although the visual world is varied, it nevertheless has ubiquitous structure. Structured factors, such as scale, admit clear theories and efficient representation design. Unstructured factors, such as what makes a cat look like a cat, are too complicated to model analytically, requiring free-form representation learning. How can recognition harness structure without restraining the representation?

Free-form representations are structure-agnostic, making them general, but not exploiting structure is computationally and statistically inefficient. Structured representations like steerable filtering (Freeman & Adelson, 1991; Simoncelli & Freeman, 1995; Jacobsen et al., 2016) , scattering (Bruna & Mallat, 2013; Sifre & Mallat, 2013) , and steerable networks (Cohen & Welling, 2017) are efficient but constrained to the chosen structures. We propose a new, semi-structured compositional filtering approach to blur the line between free-form and structured representations and learn both. Doing so learns local features and the degree of locality.

Free-form filters, directly defined by the parameters, are general and able to cope with unknown variations, but are parameter inefficient. Structured factors, such as scale and orientation, are enumerated like any other variation, and require duplicated learning across different layers and channels. Nonetheless, end-to-end learning of free-form parameters is commonly the most accurate approach to complex visual recognition tasks when there is sufficient data.

Structured filters, indirectly defined as a function of the parameters, are theoretically clear and parameter efficient, but constrained. Their effectiveness hinges on whether or not they encompass the true structure of the data. If not, the representation is limiting, and subject to error. At least, this is a danger when substituting structure to replace learning.

We compose free-form and structured filters, as shown in Figure 1 , and learn both end-to-end. Freeform filters are not constrained by our composition. This makes our approach more expressive, not less, while still able to efficiently learn the chosen structured factors. In this way our semi-structured networks can reduce to existing networks as a special case. At the same time, our composition can learn different receptive fields that cannot be realized in the standard parameterization of free-form filters. Adding more free-form parameters or dilating cannot learn the same family of filters. Figure 2 offers one example of the impracticality of architectural alternatives. 

/ + g f H j U M n G q G W + y W M a 6 E 1 D D p V C 8 i Q I l 7 y S a 0 y i Q v B 2 M b 2 Z + + 4 l r I 2 L 1 g J O E + x E d K h E K R t F K n b D f w x F H 2 i 9 X 3 K o 7 B 1 k l X k 4 q k K P R L 3 / 1 B j F L I 6 6 Q S W p M 1 3 M T 9 D O q U T D J p 6 V e a n h C 2 Z g O e d d S R S N u / G x + 7 5 S c W W V A w l j b U k j m 6 u + J j E b G T K L A d k Y U R 2 b Z m 4 n / e d 0 U w 2 s / E y p J k S u 2 W B S m k m B M Z s + T g d C c o Z x Y Q p k W 9 l b C R l R T h j a i k g 3 B W 3 5 5 l b R q V e + i W r u / r N R v 8 z i K c A K n c

A 4 e X E E d 7 q A B T W A g 4 R l e 4 c 1 5 d F 6 c d + d j 0 V p w 8 p l j + A P n 8 w c f w J A I < / l a t e x i t > g ⌃ < l a t e x i t s h a 1 _ b a s e 6 4 = " A G r X z x 9 h t 3 More free-form parameters require more data to learn and the maximum size is still bounded.

B H t 0 C 1 B c E a S i 4 v Z L 0 = " > A A A B 7 3 i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R b B U 9 m t B T 0 W R P B Y 0 X 5 A u 5 R s m t 2 G J t k 1 y Q p l 6 Z / w 4 k E R

Q E W y s 1 I 0 G / X s W C T w o V 9 y q O w d a J V 5 O K p C j O S h / 9 Y c x S Q W V h n C s d c 9 z E + N n W B l G O J 2 W + q m m C S Z j H N G e p R I L q v 1 s f u 8 U n V l l i M J Y 2 Z I G z d X f E x k W W k 9 E Y D s F N i O 9 7 M 3 E / 7 x e a s I r P 2 M y S Q 2 V Z L E o T D k y M Z o 9 j 4 Z M U W L 4 x B J M F L O 3 I j L C C h N j I y r Z E L z l l 1 d J u 1 b 1 L q q 1 u 3 q l c Z P H U Y Q T O I V z 8 O A S G n A L T W g B A Q 7 P 8 A p v

Our contributions include: (1) composing Gaussian filtering and free-form filtering to bridge classic ideas for scale-space representation design and current practices for representation learning, (2) exploring a variety of receptive fields that our approach can learn, and (3) adapting receptive fields during inference with accurate and efficient dynamic Gaussian structure.

2. RELATED WORK:

Composing structured Gaussian filters with free-form learned filters draws on structured filter design and representation learning. Our work is inspired by the transformation invariance of scale-space (Lindeberg, 1994) , the parsimony of steerable filtering (Freeman & Adelson, 1991; Perona, 1995; Bruna & Mallat, 2013; Cohen & Welling, 2017) , and the adaptivity of dynamic inference (Olshausen et al., 1993; Jaderberg et al., 2015; De Brabandere et al., 2016; Dai et al., 2017) . Analysis that the effective receptive field size of deep networks is limited (Luo et al., 2016) , and is only a fraction of the theoretical size, motivates our goal of learning and adapting unbounded receptive field sizes and varied receptive field shapes.

Transformation Invariance Gaussian scale-space and its affine extension connect covariance to spatial structure for transformation invariance (Lindeberg, 1994) . We jointly learn structured transformations via Gaussian covariance and features via free-form filtering. Enumerative methods cover a set of transformations, rather than learning to select transformations: image pyramids (Burt & Adelson, 1983 ) and feature pyramids (Kanazawa et al., 2014; Shelhamer et al., 2017; Lin et al., 2017) cover scale, scattering (Bruna & Mallat, 2013) covers scales and rotations, and steerable networks (Cohen & Welling, 2017) cover discrete groups. Our learning and inferring covariance relates to scale selection (Lindeberg, 1998) , as exemplified by the scale invariant feature transform (Lowe, 2004) . Scale-adaptive convolution (Zhang et al., 2017) likewise selects scales, but without our Gaussian structure and smoothness.

Steering Steering indexes a continuous family of filters by linearly weighting a structured basis, such as Gaussian derivatives. Steerable filters (Freeman & Adelson, 1991) index orientation and deformable kernels (Perona, 1995) index orientation and scale. Such filters can be stacked into a deep, structured network (Jacobsen et al., 2016) . These methods have elegant structure, but are constrained to it. We make use of Gaussian structure, but keep generality by composing with free-form filters.

Dynamic Inference Dynamic inference adapts the model to each input. Dynamic routing (Olshausen et al., 1993) , spatial transformers (Jaderberg et al., 2015) , dynamic filter nets (De Brabandere et al., 2016) , and deformable convolution (Dai et al., 2017) are all dynamic, but lack local structure. We incorporate Gaussian structure to improve efficiency while preserving accuracy.

Proper signal processing, by blurring when downsampling, improves the shift-equivariance of learned filtering (Zhang, 2019) . We reinforce these results with our experiments on blurred dilation, to complement their focus on blurred stride. While we likewise blur, and confirm the need for smoothing to prevent aliasing, our focus is on how to jointly learn and compose structured and free-form filters.

3. A CLEAR REVIEW OF BLURRING:

We introduce our chosen structured filters first, and then compose them with free-form filters in the next section. While the Gaussian and scale-space ideas here are classic, our end-to-end optimized composition and its use for receptive field learning are novel.

3.1. GAUSSIAN STRUCTURE:

The choice of Gaussian structure determines the filter characteristics that can be represented and learned. For learning, it is differentiable, low-dimensional for parameter efficiency, and expressive enough for different sizes and shapes. For signal processing, it is smooth and computationally efficient. In particular, the Gaussian has these attractive properties for our purposes:

• shift-invariance for convolutional filtering,

• normalization to preserve input and gradient norms for stable optimization,

• separability to reduce computation by replacing a 2D filter with two 1D filters,

• and cascade smoothing from semi-group structure to decompose filtering into smaller, cumulative steps.

In fact, the Gaussian is the unique filter satisfying these and further scale-space axioms (Koenderink, 1984; Babaud et al., 1986; Lindeberg, 1994) .

The Gaussian kernel in 2D is G(x; Σ) = 1 2π √ det Σ e -x T Σ -1 x/2 for input coordinates x and covari- ance Σ ∈ R 2×2 , a symmetric positive-definite matrix.

The structure of the Gaussian is controlled by its covariance Σ. Note that we are concerned with the spatial covariance, where the coordinates are considered as random variables, and not the covariance of the feature dimensions. Therefore the elements of the covariance matrix are σ 2 y , σ 2 x for the y, x coordinates and ρ for their correlation. The standard, isotropic Gaussian has identity covariance [ 1 0 0 1 ]. There is progressively richer structure in spherical σ 2 0 0 σ 2 , diagonal

σ 2 y 0 0 σ 2 x

, and full

σ 2 y ρ ρ σ 2

x covariances which can represent scale, aspect, and orientation in one, two, or three parameters.

Selecting the right covariance yields invariance to a given spatial transformation (Lindeberg, 1994) .

We leverage this transformation property of Gaussians to learn receptive fields in Section 4.1 and dynamically adapt them for locally invariant filtering in Section 4.2.

From the Gaussian kernel G(x, Σ) we instantiate a Gaussian filter g Σ (•) in the standard way: (1) evaluate the kernel at the coordinates of the filter coefficients and (2) renormalize by the sum to correct for this discretization. We decide the filter size according to the covariance by setting the half size = 2σ in each dimension. This covers ± 2σ to include 95% of the true density no matter the covariance. (We found that higher coverage did not improve our results.) Our filters are always odd-sized to keep coordinates centered.

3.2. COVARIANCE PARAMETERIZATION & OPTIMIZATION:

The covariance Σ is symmetric positive definite, requiring proper parameterization for unconstrained optimization. We choose the log-Cholesky parameterization (Pinheiro & Bates, 1996) for iterative optimization because it is simple and quick to compute: Σ = U U for upper-triangular U with positive diagonal. We keep the diagonal positive by storing its log, hence log-Cholesky, and exponentiating when forming Σ. (See Pinheiro & Bates (1996) for a primer on covariance parameterization.)

Here is an example for full covariance Σ with σ 2 y , σ 2 x for the y, x coordinates and ρ for their correlation: 

σ 2 y ρ ρ σ 2 x ← +1 -2 -2 +8 = +1 +0 -2 +2 +1 -2 +0 +2 = (log(1), -2, log(2)).

3.3. LEARNING TO BLUR:

As a pedagogical example, consider the problem of optimizing covariance to reproduce an unknown blur. That is, given a reference image and a blurred version of it, which Gaussian filter causes this blur? Figure 3 shows such an optimization: from an identity-like initialization the covariance parameters quickly converge to the true Gaussian.

Given the full covariance parameterization, optimization controls scale, aspect, and orientation. Each degree of freedom can be seen across the iterates of this example. Had the true blur been simpler, for instance spherical, it could still be swiftly recovered in the full parameterization.

Notice how the size and shape of the filter vary over the course of optimization: this is only possible through structure. For a Gaussian filter, its covariance is the intrinsic structure, and its coefficients follow from it. The filter size and shape change while the dimension of the covariance itself is constant. Lacking structure, free-form parameterization couples the number of parameters and filter size, and so cannot search over size and shape in this fashion.

4. COMPOSING FREE-FORM & GAUSSIAN FILTERING:

Deep visual representations are made by composing convolutions to learn rich features and receptive fields, which characterize the spatial extent of the features. Although each filter might be small, and relatively simple, their composition can represent and learn large, complex receptive fields. For instance, a stack of two 3 × 3 filters is effectively 5 × 5 but with fewer degrees of freedom (2 • 3 2 vs. 5 2 ). Composition therefore induces factorization of the representation, and this factorization determines the generality and efficiency of the representation.

Our semi-structured composition factorizes the representation into spatial Gaussian receptive fields and free-form features. This composition is a novel approach to making receptive fields differentiable, low-dimensional, and decoupled from the number of parameters. Our approach jointly learns the structured and free-form parameters while guaranteeing proper sampling for smooth signal processing.

Purely free-form filters cannot learn size and shape in this way: size is bounded by the number of parameters and shape depends on all of the parameters. Purely structured filters, restricted to Gaussians and their derivatives for instance, lack the generality of free-form filters. Our factorization into structured and free-form filters is efficient for the representation, optimization, and inference of receptive fields without sacrificing the generality of features.

Receptive field size is a key design choice in the architecture of fully convolutional networks for local prediction tasks (Shelhamer et al., 2017) . The problem of receptive field design is encountered with each new architecture, dataset, or task. Trying candidate receptive fields by enumeration is costly, in effort and computation, whether by manual or automated model search (Zoph & Le, 2017; Kandasamy et al., 2018; Liu et al., 2019) . By making this choice differentiable, we show that learning can adjust to changes in the architecture and data in Section 5.2. Optimizing our semi-structured filters in effect searchs over receptive fields. Our composition helps relieve the burden of architecture design by relaxing the receptive field from a discrete decision into a continuous optimization.

4.1. CONVOLUTIONAL COMPOSITION OF STATIC GAUSSIANS:

Our composition f θ • g Σ combines a free-form f θ with a structured Gaussian g Σ . This combination reduces to convolution, and so it inherits the efficiency of aggressively tuned convolution routines. Since convolution is associative, filtering of an input I decomposes into two steps of convolution by

I * (g Σ * f θ ) = I * g Σ * f θ .

This decomposition has computational advantages. The Gaussian step can be done by specialized filtering that harnesses separability, cascade smoothing, and other structure. Memory can be spared by only keeping the covariance parameters and instantiating the filters as needed. Each compositional filter can always be explicitly formed by g Σ * f θ for visualization (see Figure 1 ) or other analysis.

Both θ and Σ are differentiable for end-to-end learning.

σ → 0 σ < 1 σ → ∞ Figure 4:

The identity is recovered by a delta as variance goes to zero. Small variance gives a good initialization near the identity. Average pooling is approximated as variance goes to infinity.

There are useful special cases of the Gaussian when considering covariance optimization as differentiable model search. Figure 4 illustrates covariances for the identity, initialization, and pooling. Because the identity is included in the limit, our models can recover standard networks that lack our composition. By initializing near the identity, we are able to augment pre-trained networks without interference, and let learning decide whether or not to make use of structure. Blurring for Smooth Signal Processing Even without learning the covariance, blurring can improve dilation by avoiding aliasing. Figure 5 shows the effect of blurring dilation. Smoothing when subsampling is a fundamental technique in signal processing (Oppenheim & Schafer, 2009) , so we merely note this fix as a simple alternative to the careful re-engineering of dilated architectures (Yu et al., 2017; Wang et al., 2018) .

Compound Gaussian Structure Gaussian filters have a special compositional structure: cascade smoothing. Composing a Gaussian g Σ with a Gaussian g Σ is still Gaussian with covariance Σ + Σ . This lets us efficiently assemble compound receptive fields made of multiple Gaussians. Center-surround (Kuffler, 1953) receptive fields, which boost contrast, can be realized by such a combination as Difference-of-Gaussian (Rodieck & Stone, 1965) (DoG) filters, which subtract a larger Gaussian from a smaller Gaussian. Our joint learning of their covariances tunes the contrastive context of the receptive field, extending (Ding et al., 2018) which learns contrastive filters with fixed receptive field sizes.

Design Choices:

We cover the design choices involved in the application of our composition. As a convolutional composition, it can augment any convolution layer in the architecture. We focus on including our composition in late, deep layers to show the effect without much further processing. We add compositional filtering to the output and decoder layers of fully convolutional networks because the local tasks they address rely on the choice of receptive fields.

Having decided where to compose, we must decide how much structure to include. We explore minimal structure, where one Gaussian is shared across the free-form filters of a layer, and spatially dynamic structure, where the receptive field at each location varies with the input.

4.2. DEFORMABLE COMPOSITION OF DYNAMIC GAUSSIANS:

Dynamic inference replaces static, global parameters with local parameters, inferred from the input, to adapt to these variations. There are two routes to dynamic Gaussian structure: local filtering and deformable sampling. Local filtering has a different filter kernel for each position, as done by dynamic filter networks (De Brabandere et al., 2016) . This ensures exact filtering for dynamic Gaussians, but is too computationally demanding for large-scale recognition networks. Deformable sampling adjusts the position of filter taps by dynamic offsets, as done by deformable convolution (Dai et al., 2017) . We exploit deformable sampling for the sparse approximation of dynamic Gaussians. (e) diagonal Gauss.

(f) full Gauss.

Figure 6 : Gaussian deformation (c-f) structures dynamic receptive fields by controlling the sampling points (blue) according to the covariance. Its lowdimensionality is less general but more efficient for learning and inference than free-form deformation (b) by Dai et al. (2017) , which requires more parameters.

We constrain deformable sampling to Gaussian structure by setting the sampling points according to covariance. Figure 6 illustrates Gaussian deformations. Our default deformation approximates the standard Gaussian. We consider the same progression of spherical, diagonal, and full covariance for dynamic structure. This low-dimensional structure differs from the high degrees of freedom in a dynamic filter network, which sets free-form filter parameters, and deformable convolution, which sets free-form offsets. Our Gaussian deformation requires only a small, constant number of covariance parameters independent of the sampling resolution and the kernel size k, while deformable convolution has constant resolution and requires 2k 2 offset parameters for a k × k filter.

To infer the local covariances we learn a convolutional regressor, which is simply a convolutional filter. The inferred covariances then determine the receptive fields of the the following free-form filters. The low-dimensional structure of our dynamic parameters makes this regressor more efficient than the regressor for free-form deformation, as it only has three outputs for each full covariance, or even just one for each spherical covariance. Since the covariance is differentiable, the regression is learned endto-end from the task loss without further supervision.

5. EXPERIMENTS:

We experiment with the local recognition task of semantic segmentation, because our method learns local receptive fields. As a recognition task, semantic segmentation requires a balance between local scope, to infer where, and global scope, to infer what.

Data CityScapes (Cordts et al., 2016 ) is a challenging dataset of varied urban scenes from the perspective of a car-mounted camera. We follow the standard training and evaluation protocols and train/validation splits, with 2, 975 finely-annotated training images and 500 validation images. We score results by the common intersection-over-union metric on the validation set. We evaluate the network itself without post-processing, test-time augmentation, or other accessories to isolate the effect of receptive field learning.

Architecture and Optimization For backbones we choose strong fully convolutional networks derived from residual networks (He et al., 2016) . Dilated residual nets (DRN) (Yu et al., 2017) have high resolution and receptive field size through dilation. Deep layer aggregation (DLA) (Yu et al., 2018) fuses layers by hierarchical and iterative skip connections. We also define a ResNet-34 backbone as a simple alternative for ablations and exploratory experiments. These are representative of common architectural patterns in state-of-the-art fully convolutional networks.

We train our models by stochastic gradient descent for 240 epochs with momentum 0.9, batch size 16, and weight decay 10 -4 . Training follows the "poly" learning rate schedule (Chen et al., 2018; Zhao et al., 2017) with initial rate 0.01. The input images are cropped to 800 × 800 and augmented by random scaling, random rotation, and random color distortions as in (Howard, 2013) . We train with synchronized, in-place batch normalization (Rota Bulò et al., 2018) . For fair comparison, we reproduce the DRN and DLA baselines in our same setting, which improves on their reported results.

Baselines The chosen DRN and DLA architectures are strong methods on their own, but they can be further equipped for learning global spatial transformations and local deformations. Spatial transformer networks (Jaderberg et al., 2015) and deformable convolution (Dai et al., 2017) Table 3 : Blurring dilation helps slightly.

5.1. LEARNING SEMI-STRUCTURED FILTERS:

Augmenting Existing Architectures See Table 1 for the accuracies of the architectures, baselines, and our filtering. We augment the last output stage of DRN-A with a single instance of our composition. We augment the decoder of DLA with ten instances of our composition, one at each merge layer. Optimization is end-to-end. Our composition by convolution improves by 1-2 points.

Note that these architectures are already agressively-tuned, which required significant model search and engineering effort. Our composition is still able to deliver improvement through learning without further engineering. In the next subsection, we show that joint optimization of our composition assists model search when the chosen architecture is suboptimal.

How to Compose:

We can compose with a Gaussian structured filter by blurring alone or blurring and resampling. Blurring and resampling first blurs with the Gaussian, and then transforms the sampling points for the following filtering according to the covariance. Blurring and resampling by Gaussian covariance can be interpreted as a smooth, circular extension of dilated convolution (Chen et al., 2015; Yu & Koltun, 2016) or as a smooth, affine restriction of deformable convolution (Dai et al., 2017) . As either can be learned end-to-end, we experiment with both in Table 2 . From this comparison we choose blurring and resampling for the remainder of our experiments.

Blurred Dilation To isolate the effect of blurring without learning, we smooth dilation with a blur proportional to the dilation rate. CCL (Ding et al., 2018) and ASPP (Chen et al., 2018) are carefully designed dilation architectures for context modeling, but neither blurs before dilating. Improvements from blurred dilation are reported in Table 3 . Although the gains are small, this establishes that smoothing can help. This effect should only increase with dilation rate.

5.2. DIFFERENTIABLE RECEPTIVE FIELD SEARCH:

Our composition turns choosing receptive fields into a task for optimization, instead of design. Table 4 shows how optimization counteracts the reduction of the architectural receptive field size and the enlargement of the input. These controlled experiments, while simple, reflect a realistic lack of knowledge in practice: for a new architecture or dataset, the right design is unknown.

For these experiments we include our composition in the last stage of the network. We compare fine-tuning only the last stage and end-to-end optimization. End-to-end optimization of our difference of Gaussians significantly reduces the drop in accuracy across scale shifts.

In the extreme, we can do structural fine-tuning by including our composition in a pre-trained network and only optimizing the covariance. When fine-tuning the structure alone, optimization either reduces the Gaussian to a delta, doing no harm, or slightly enlarges the receptive field, giving a one point boost. Therefore the special case of the identity, as explained in Figure 4 , is learnable in practice.

5.3. DYNAMIC GAUSSIAN STRUCTURE:

Dynamic inference of the covariance adaptively adjusts receptive fields to vary with the input. In these experiments we choose spherical covariance with a single degree of freedom for scale. 5 . Gaussian structure improves efficiency while preserving accuracy.

Our results show spherical Gaussian deformation can suffice to achieve equal accuracy as general, free-form deformation. Including further degrees of freedom by diagonal and full covariance does not give further improvement on this task and data. As scale is a ubiquitous transformation in the distribution of natural images, dynamic scale inference might suffice to cope with many variations.

6. CONCLUSION:

Composing structured Gaussian and free-form filters makes receptive fields differentiable for direct optimization of the degree of locality. Through receptive field learning, our semi-structured parameters do by gradient optimization what current free-form architectures have done by discrete design. That is, in our parameterization changes in structured weights would require changes in free-form architecture.

Our method learns local receptive fields. While we have focused on locality in space, the principle is more general, and extends to locality in time and other dimensions.

Factorization of this sort points to a reconciliation of structure and learning, through which known structure is respected and unknown detail is learned freely.

:



Figures and Tables:

Figure 1: We compose free-form filters f θ and structured Gaussian filters g Σ by convolution * to define a more general family of filters than either alone. Our composition makes filter size differentiable for end-to-end learning.

r / 4 d b / 4 b 0 3 Y P 2 v p g 4 P H e D D P z g o Q z b V z 3 2 y m s r W 9 s b h W 3 S z u 7 e / s H 5 c O j t o 5 T R W i L x D x W 3 Q B r y p m k L c M M p 9 1 E U S w C T j v B + H r m d 5 6 o 0 i y W D 2 a S U F / g S L K

Figure 2: Our composition does not reduce to dilation or more free-form parameters. Dilation is sparse, causing artifacts, and cannot be learned. More free-form parameters require more data to learn and the maximum size is still bounded.

Figure 3: Recovering an unknown blur by optimizing covariance. Gradient optimization of the structured parameters Σ quickly converges to the true Gaussian. Although this is a simple example, it shows how the Gaussian can represent scale, aspect, and orientation with variable filter size.

Figure 5: Blurring prevents aliasing by smoothing dilated filters to respect the sampling theorem.

Figure 7: Qualitative results for dynamic scale inference: (a) inputs, (b) truths, (c) outputs, and (d) scale estimates where small is blue/dark and large is yellow/bright. The scale estimates exhibit structure: coherent segments and boundaries between them can be seen.Qualitative results for dynamic Gaussian structure are shown in Figure7. The local covariances reflect scale structure in the input and the output segmentation. Quantitative results compare with free-form deformation in Table5. Gaussian structure improves efficiency while preserving accuracy.

learn dynamic global/local transformations respectively. Spatial transformers serve as a baseline for structure, because they are restricted to a parametric class of transformations. Deformable convolution serves as a baseline for local, dynamic inference without structure. For comparison in the static setting, we simplify both methods to instead learn static transformations. Learning our composition improves the accuracy of careful designs.

Table content: <table><tr><td>method</td><td>IU</td><td/><td/></tr><tr><td>DRN-A</td><td>72.4</td><td>method</td><td>IU</td></tr><tr><td colspan="2">+ Extra Conv. + STN (static) + Deformable (static) 72.2 72.9 70.5 + Composition (ours) 73.5 + CCL 73.1 + DoG (ours) 74.1</td><td colspan="2">ResNet-34 + Blur + Blur-Resample + DoG Blur + DoG Blur-Resample 71.4 64.8 66.3 68.1 70.3 DRN-A 72.4 + Blur 72.2</td><td>method DRN-A w/ CCL + Blur w/ ASPP 74.1 IU 72.4 73.1 74.0 + Blur 74.3</td></tr><tr><td>DLA-34</td><td>76.1</td><td>+ Blur-Resample</td><td>73.5</td></tr><tr><td colspan="2">+ Composition (ours) 78.2</td><td/><td/></tr><tr><td/><td/><td colspan="2">Table 2: It is better to compose</td></tr><tr><td/><td/><td colspan="2">with resampling than without.</td></tr></table>

Optimizing receptive fields with our composition helps adjust to the architecture and data.

Table content: <table><tr><td/><td/><td/><td/><td>Cityscapes Validation</td><td/><td/></tr><tr><td colspan="4">method DRN-A No Dilation: Smaller Receptive Field params epoch many 240 72.4 IU ResNet-34 many 240 64.8 -7.6 ∆ 0 + Composition some +20 68.1 -4.6 + DoG some +20 68.9 -3.5 . . . End-to-End many 240 71.4 -0.8 and 2× Enlarged Input</td><td>method DRN-A + Gauss. Def. (ours) + Free-form Def. ResNet-34 + Gauss. Def. (ours) + Free-form Def.</td><td>dyn.?</td><td>dyn. params -1 2k 2 -1 2k 2</td><td>IU 72.4 76.6 76.6 64.8 74.2 75.1</td></tr><tr><td>ResNet-34</td><td>many</td><td>240</td><td>56.2 -16.2</td><td>Cityscapes Test</td><td/><td/></tr><tr><td>+ Composition</td><td>some</td><td>+20</td><td>57.8 -14.6</td><td>DRN-A</td><td/><td>-</td><td>71.2</td></tr><tr><td>+ DoG</td><td>some</td><td>+20</td><td>62.7 -9.7</td><td>+ Gauss. Def. (ours)</td><td/><td>1</td><td>74.3</td></tr><tr><td>. . . End-to-End</td><td>many</td><td>240</td><td>66.5 -5.9</td><td>+ Free-form Def.</td><td/><td>2k 2</td><td>73.6</td></tr></table>

Gaussian deformation improves computational efficiency and rivals free-form accuracy.

Table content: <table/>

Peer review (iclr2020):
Summary: - key problem: improved visual representation learning with limited increase in parameters by leveraging Gaussian structure; - contributions: 1) compose Gaussian blurs and free-form convolutional filters in an end-to-end differentiable fashion, 2) showing that learning the covariance enables a factorized parameter-efficient representation covering wide and flexibly structured filters, 3) experiments on CityScapes showing the proposed layers can help improve semantic segmentation performance for different architectures (DRN, DLA, and ResNet34). Recommendation: weak reject Key reason 1: mismatch between the generality of the claims and experiments. - Learning to adapt and optimize receptive fields successfully would be a great fundamental improvement to CNN architectures. Experiments are done on a single dataset for a single task, which seems insufficient to support the generality of the approach and claims in the submission. I would recommend using other datasets (e.g., COCO) and tasks (e.g., object detection, instance segmentation, depth estimation/completion), where the benefits of the approach could be demonstrated more broadly and clearly (including its inherent trade-offs). - The improved efficiency (one of the main claims) is only assessed on the number of parameters, which is a direct consequence of the parametrization. Is it significant at the scale of the evaluated architectures? Does it result in runtime performance benefits? If it is indeed a useful structural inductive bias, does it result in improved few-shot generalization performance or less overfitting? Does it enable learning deeper networks on the same amount of data? - Why modifying only later layers in the architecture (end of 4.1)? It seems that early layers would make sense too, as it is where most of the downsampling happens. Key reason 2: lack of clarity and details. - Section 1 and the beginning of section 4 are repetitive and verbose; in particular, Sections 4.1 and 4.2 would benefit from less textual descriptions replaced by more concise mathematical formula (simpler in this case), especially in order to know the details behind the methods compared in Tables 1-2-3. - Overall, the paper could contain less text describing the hypothetical advantages of the method and the basic preliminaries (section 3), to focus more on the method itself, its details and evaluated benefits. In particular, the dynamic part (section 4.2) is unclear and the method is mostly described in one sentence: *To infer the local covariances we learn a convolutional regressor, which is simply a convolutional filter.* Another example of the lack of details is *many* vs. *some* in the *params* column of Table 4. - There is also a missed opportunity to provide compelling feature visualizations and qualitative experiments (beyond Fig. 7). For instance, what are the typical covariances learned? What are the failure modes that the proposed modifications address, in particular w.r.t. thin structures and boundaries that are typical hard cases for semantic segmentation and where blurring might be counterproductive? Additional Feedback: - missing reference: Learning Receptive Field Size by Learning Filter Size, Lee et al, WACV*19; - missing reference (w.r.t. local filtering): Pixel-Adaptive Convolutional Neural Networks, Su et al, CVPR*19