Blurring Structure and Learning to Optimize and Adapt Receptive Fields
Authors: Evan Shelhamer
Dequan Wang
Trevor Darrell
Venue:  (2020)
Keywords: scale
deep learning
dynamic inference
fully convolutional

Abstract: The visual world is vast and varied, but its variations divide into structured and unstructured factors. We compose free-form filters and structured Gaussian filters, optimized end-to-end, to factorize deep representations and learn both local features and their degree of locality. In effect this optimizes over receptive field size and shape, tuning locality to the data and task. Our semi-structured composition is strictly more expressive than free-form filtering, and changes in its structured parameters would require changes in architecture for standard networks. Dynamic inference, in which the Gaussian structure varies with the input, adapts receptive field size to compensate for local scale variation. Optimizing receptive field size improves semantic segmentation accuracy on Cityscapes by 1-2 points for strong dilated and skip architectures and by up to 10 points for suboptimal designs. Adapting receptive fields by dynamic Gaussian structure further improves results, equaling the accuracy of free-form deformation while improving efficiency.

Under review as a conference paper at ICLR 2020
BLURRING STRUCTURE AND LEARNING TO OPTIMIZE AND ADAPT RECEPTIVE FIELDS
Anonymous authors Paper under double-blind review
ABSTRACT
The visual world is vast and varied, but its variations divide into structured and unstructured factors. We compose free-form filters and structured Gaussian filters, optimized end-to-end, to factorize deep representations and learn both local features and their degree of locality. effect this optimizes over receptive field size and shape, tuning locality the data and task. Our semi-structured composition is strictly more expressive than free-form filtering, and changes in its structured parameters would require changes architecture for standard networks. Dynamic inference, in which the Gaussian structure varies with the input, adapts receptive field size to compensate for local scale variation. Optimizing receptive field size improves semantic segmentation accuracy on Cityscapes by 1-2 points for strong dilated and skip architectures and by up to 10 points for suboptimal designs. Adapting receptive fields by dynamic Gaussian structure further improves results, equaling the accuracy of free-form deformation while improving efficiency.
INTRODUCTION
Although the visual world varied, it nevertheless has ubiquitous structure. Structured factors, such as scale, admit clear theories and efficient representation design. Unstructured factors, such as what makes a cat look like a cat, are too complicated to model analytically, requiring free-form representation learning. How can recognition harness structure without restraining the representation?
Free-form representations are structure-agnoster making them general, but not exploiting structure computationally and statistically inefficient. Structured representations like steerable filtering (Freeman Adelson| 1991 | Simoncelli & Freeman 1995 Jacobsen et al.) (0016), scattering (Bruna & Mallat 2013; Sifre & Mallat 2013), and steerable networks (Cohen & Welling 2017) are efficient but constrained to the chosen structures. We propose a new, stru-stteeturre compositional filtering approach blur the line between free-form and structured representations and learn both. Doing so learns local features and the degree of locality.
Free-form filters, directly defined by the parameters, are general and able to cope with unknown variations, but are parameter inefficient. Structured factors, such as scale and orientation, are enumerated like any other variation, and require duplicated learning across different layers and channels. Nonetheless, end-to-end learning of free-form parameters is commonly the most accurate approach to complex visual recognition tasks when there sufficient data.
Structured filters, indirectly defined as a function of the parameters, are theoretically clear and parameter efficient, but constrained. Their effectiveness hinges on whether or not they encompass the true structure of the data. If not, the representation is limiting, and subject to error. At least, this is danger when substituting structure to replace learning.
We compose free-form and structured filters, as shown in Figure | and learn both end-to-end. Free- form filters are not constrained by our composition. This makes our approach more expressive, not less, while still able to efficiently learn the chosen structured factors. In this way our semi-structured networks can reduce to existing networks as a special case. At the same time, our composition can learn different receptive fields that cannot be realized in the standard parameterization of free-form filters. Adding more free-form parameters or dilating cannot learn the same family of filters. Figure] 2 offers one example of the impracticality of architectural alternatives.

---

Under review as a conference paper at ICLR 2020
fo
0 RTx7
7
7x7+3
7x7
21 21
fø
ΣE
P
P
9>:
composition
dilation
large kereel
fo
Figure 2:
Our composition does not reduce to dilation or more free-form parameters. Dilation sparse, causing artifacts, and cannot be learned. More free-form parameters require more data to learn and the maximum size is still bounded.
Figure We compose free-form filters Se and structured Gaussian filters g) by convolution * to define a more general family of filters than either alone. Our composition makes filter size differentiable for end-to-end learning.
Our contributions include: (1) composing Gaussian filtering and free-form filtering to bridge classic ideas for scale-space representation design and current practices for representation learning, (2) exploring variety of receptive fields that approach leam, and (3) adapting receptive fields during inference with accurate and efficient dynamic Gaussian structure.
RELATED WORK
Composing structured Gaussian filters with free-form learned filters draws structured filter design and representation learning. Our work is inspired by the transformation invariance of scale-space (Lindeberg) 1994). the parsimony of steerable filtering (Freeman & Adelson 1991; Perona 1995)
Bruna & Mallati [013 Cohen & Welling (017). and the adaptivity of dynamic inference (Olshausen al.] 1993 1993]99993 Jaderberg
2015] | De Brabandere et al. [2066] Dai et al. 2017). Analysis that the
effective receptive field size of deep networks is limited (Luo et al. 2016). and is only fraction of
the theoretical size, motivates our goal of learning and adapting unbounded receptive field sizes and
varied receptive field shapes.
Transformation Invariance Gaussian scale-space and affine extension connect covariance to spatial structure for transformation invariance (Lindeberg) 1994). We jointly learn structured transfor- mations via Gaussian covariance and features via free-form filtering. Enumerative methods cover set of transformations, rather than learning to select transformations: image pyramids (Burt & Adelson 1983) and feature pyramids (Kanazawa et al. 2014 Shelhamer et al., 2017] Lin et al.) (017) cover scale, scattering (Bruna & Mallat 2013) covers scales and rotations, and steerable networks (Cohen & Welling (017) cover discrete groups. Our learning and inferring covariance relates to scale selection (Lindeberg 1998), exemplified by the scale invariant feature transform (Lowe) 2004).
Scale-adaptive convolution (Zhang et al. 2017) likewise selects scales, but without our Gaussian structure and smoothness.
Steering Steering indexes a continuous family of filters by linearly weighting a structured basis, such as Gaussian derivatives. Steerable filters (Freeman & Adelson 1991) index orientation and deformable kernels (Perona 1995) index orientation and scale. Such filters can be stacked into deep, structured network (Jacobsen al. 2016). These methods have elegant structure, but are constrained to it. We make use of Gaussian structure, but keep generality by composing with free-form filters.
Dynamic Inference Dynamic inference adapts the model each input. Dynamic routing (Olshausen et al. 1993), spatial transformers (Jaderberg al.) 20155, dynamic filter nets (De Brabandere et al. 2016), and deformable convolution (Dai et al. 2017) are all dynamic, but lack local structure. We incorporate Gaussian structure to improve effiiency while preserving accuracy.
Proper signal processing, by blurring when downsampling, improves the shift-equivariance of learned filtering Zhang 2019). We reinforce these results with experiments blurred dilation, to complement their focus on blurred stride. While we likewise blur, and confirm the need for smoothing prevent aliasing, focus is how to jointly learn and compose structured and free-form filters.
2

---

Under review as a conference paper at ICLR 2020
3 A CLEAR REVIEW OF BLURRING
We introduce chosen structured filters first, and then compose them with free-form filters in the next section. While the Gaussian and scale-space ideas here are classic, our end-to-end optimized composition and its use for receptive field learning novel.
3.1 GAUSSIAN STRUCTURE
The choice of Gaussian structure determines the filter characteristics that can be represented and learned. For learning, is differentiable, low-dimensional for parameter efficiency, and expressive enough for different sizes and shapes. For signal processing. it is smooth and computationally efficient. In particular, the Gaussian has these attractive properties for our purposes:
for convolutional filtering,
normalization to preserve input and gradient norms for stable optimization,
separability to reduce computation by replacing a 2D filter with two ID filters,
and cascade smoothing from semi-group structure to decompose filtering into smaller, cumulative steps.
In fact, the Gaussian is the unique filter satisfying these and further scale-space axioms (Koenderink 1984 Babaud et 1986 Lindeberg 1994).
The Gaussian kernel in 2D is G(x: Σ) de 'xxx2 for input coordinates x and covari-
ance 2 _2x2, symmetric positive-definite matrix.
The structure of the Gaussian is controlled by its covariance Σ. Note that we are concerned with the spatial covariance, where the coordinates are considered as random variables, and not the covariance of the feature dimensions. Therefore the elements of the covariance matrix are og o² for the coordinates and p for their correlation. The standard, isotropic Gaussian has identity covariance 0
There is progressively richer structure in spherical [a - diagonal 1 and full " P covariances which can represent scale, aspect, and orientation in one, two, or three parameters.
0 o
o
Selecting the right covariance yields invariance given spatial transformation (Lindeberg 1994). We leverage this transformation property of Gaussians to learn receptive fields in Section 4. and dynamically adapt them for locally invariant filtering Section 4.2
From the Gaussian kernel G(x, Σ) we instantiate a Gaussian filter 95((-) in the standard way: (1) evaluate the kernel the coordinates of the filter coefficients and (2) renormalize by the sum to correct for this discretization. We decide the filter size according to the covariance by setting the half size [ in each dimension. This covers + 2o to include 95% of the true density no matter the covariance. (We found that higher coverage did not improve our results.) Our filters are always odd-sized to keep coordinates centered.
3.2 COVARIANCE PARAMETERIZATION & OPTIMIZATION
The covariance symmetric positive definite, requiring proper parameterization for unconstrained optimization. We choose the Pog-Cholesky parameterization (Pinheiro & Bates] 1996) for iterative optimization because is simple and quick to compute: = U' for upper-triangular U with positive diagonal. We keep the diagonal positive by storing its log, hence log-Cholesky, and exponentiating when forming Σ. (See/Pinherl Pinheiro & Bates (199) for primer on covariance parameterization)
Here is an example for full covariance Σ with oy o for the y, x coordinates and p for their correlation: [
p of [=1=2] [+] +0 +1 -21 (log(1). -2, log(2)). Spherical and diagonal covariance are parameterized by fixing p O and tying/untying dy. Ox-
3

---

Under review as a conference paper at ICLR 2020
argmin
gg
out
iter.
16
64
256
truth
Figure 3: Recovering unknown blur by optimizing covariance. Gradient optimization of the structured parameters Σ quickly converges to the true Gaussian. Although this is a simple example, shows how the Gaussian can represent scale, aspect, and orientation with variable filter size.
3.3 LEARNING TO BLUR
As pedagogical example, consider the problem of optimizing covariance to reproduce an unknown blur. That is, given reference image and blurred version of it, which Gaussian filter causes this blur? Figure 3 shows such an optimization: from an identity-like initialization the covariance parameters quickly converge to the true Gaussian.
Given the full covariance parameterization, optimization controls scale, aspect, and orientation. Each degree of freedom can be seen across the iterates of this example. Had the true blur been simpler, for instance spherical, it could still be swiftly recovered the full parameterization.
Notice how the size and shape of the filter vary over the course of optimization: this is only possible through structure. For a Gaussian filter, its covariance the intrinsic structure, and its coefficients follow from it. The filter size and shape change while the dimension of the covariance itself is constant. Lacking structure, free-form parameterization couples the number of parameters and filter size, and so cannot search over size and shape in this fashion.
4 COMPOSING FREE-FORM & GAUSSIAN FILTERING
Deep visual representations are made by composing convolutions to learn rich features and receptive fields, which characterize the spatial extent of the features. Although each filter might be small, and relatively simple, their composition can represent and learn large, complex receptive fields. For instance, stack of two 3 x filters is effectively 5 x 5 but with fewer degrees of freedom (2 32 vs. 5²). Composition therefore induces factorization of the representation, and this factorization determines the generality and efficiency of the representation.
Our semi-structured composition factorizes the representation into spatial Gaussian receptive fields and free-form features. This composition is novel approach to making receptive fields differentiable, low-dimensional, and decoupled from the number of parameters. Our approach jointly learns the structured and free-form parameters while guaranteeing proper sampling for smooth signal processing. Purely free-form filters cannot learn size and shape in this way: size is bounded by the number of parameters and shape depends on all of the parameters. Purely structured filters, restricted to Gaussians and their derivatives for instance, lack the generality of free-form filters. Our factorization into structured and free-form filters efficient for the representation, optimization, and inference of receptive fields without sacrificing the generality of features.
Receptive field size is a key design choice in the architecture fully convolutional networks for local prediction tasks (Shelhamer et al. (017)). The problem of receptive field design is encountered with each new architecture, dataset, or task. Trying candidate receptive fields by enumeration is costly, in effort and computation, whether by manual automated model search (Zoph & Le 2017] Kandasamy et al. 20188 Liu et al. 2019). By making this choice differentiable, we show that learning adjust to changes in the architecture and data in Section 5.2 Optimizing our semi-structured filters in effect searchs over receptive fields. Our composition helps relieve the burden of architecture design by relaxing the receptive field from discrete decision into continuous optimization.
4

---

Under review as a conference paper at ICLR 2020
4.1 CONVOLUTIONAL COMPOSITION OF STATIC GAUSSIANS
Our composition fo 92 combines a free-form fo with a structured Gaussian gg. This combination reduces convolution, and so it inherits the efficiency of aggressively tuned convolution routines. Since convolution is associative, filtering of an input decomposes into two steps of convolution by
(ge= fo) 9x fo-
This decomposition has computational advantages. The Gaussian step can be done by specialized filtering that harnesses separability, cascade smoothing, and other structure. Memory can be spared by only keeping the covariance parameters and instantiating the filters needed. Each compositional filter can always be explicitly formed by 9S Se for visualization (see Figure or other analysis.
Both I and Σ are differentiable for end-to-end learning.
There are useful special cases of the Gaussian when considering covariance optimization as differentiable model search. Figure 4 iillustrates covariances for the identity, initialization, and pooling. Because the identity is included in the limit, our models can recover standard networks that lack our composition. By initializing near the identity, we are able to augment pre-trained networks without interference, and let learning decide whether or not make use of structure.
0
1
Figure 4: The identity is recovered by delta as variance goes to zero. Small variance gives good initialization near the identity. Average pooling is approximated as variance goes to infinity.
Blurring for Smooth Signal Processing Even without leamning the covariance, blurring can improve dilation by avoiding aliasing. Figure [ shows the effect of blurring dilation. Smoothing when subsampling a fundamental technique in signal processing (Oppenheim & Schafer 2009), so we merely note this fix as a simple alternative to the careful re-engincering of dilated architectures Yu et al. 2017] Wang et al. 2018).
input
filter
output
with blur
Figure 5: Blurring prevents aliasing by smoothing dilated filters to respect the sampling theorem.
Compound Gaussian Structure Gaussian filters have
special compositional structure: cascade smoothing. Composing a Gaussian gE with a Gaussian Øx is still Gaussian with covariance Σ + Σ'. This lets us efficiently assemble compound receptive fields made of multiple Gaussians. Center-surround (Kuffler) 1953) receptive fields, which boost contrast, be realized by such combination as Difference-of -fa-o-siinnaaannnnnnaa (Rodieck & Stone 1965) (DoG) filters, which subtract a larger Gaussian from a smaller Gaussian. Our joint learning of their covariances tunes the contrastive context the receptive field, extending (Ding al. 2018) which learns contrastive filters with fixed receptive field sizes.
Design Choices We cover the design choices involved in the application of our composition. As convolutional composition, it can augment any convolution layer in the architecture. We focus on including our composition late, deep layers to show the effect without much further processing. We add compositional filtering to the output and decoder layers of fully convolutional networks because the local tasks they address rely on the choice receptive fields.
Having decided where to compose, we must decide how much structure to include. We explore minimal structure, where one Gaussian shared across the free-form filters of a layer, and spatially dynamic structure, where the receptive field at each location varies with the input.
4.2 DEFORMABLE COMPOSITION OF DYNAMIC GAUSSIANS
Dynamic inference replaces static, global parameters with local parameters, inferred from the input, to adapt to these variations. There are two routes to dynamic Gaussian structure: local filtering and deformable sampling. Local filtering has a different filter kernel for each position, as done by dynamic filter networks (De Brabandere et al. 2016). This ensures exact filtering for dynamic Gaussians, but too computationally demanding for large-scale recognition networks. Deformable sampling adjusts the position of filter taps by dynamic offsets, as done by deformable convolution (Dai et al. (017). We exploit deformable sampling for the sparse approximation of dynamic Gaussians.
5

---

Under review as a conference paper at ICLR 2020
We constrain deformable sampling to Gaussian structure by setting the sampling points according to covariance. Figure 66 illustrates Gaussian deformations. Our default deformation approximates the standard Gaussian. We con- sider the same progression of spherical, diagonal, and full covariance for dynamic structure. This oow-dimensional structure differs from the high degrees of freedom in dynamic filter network, which sets frre-form filter param- eters, and deformable convolution, which sets free-form offsets. Our Gaussian deformation requires only a small, constant number of covariance parameters independent of the sampling resolution and the kernel size k, while de- formable convolution has constant resolution and requires 2k offset parameters for & k filter.
(a)
(b)
3 3 filter
deformable
standard Gauss.
(e)
(f)
spherical Gauss.
diagonal Gauss.
full Gauss.
To infer the local covariances we learn a convolutional regressor, which is simply convolutional filter. The in- ferred covariances then determine the receptive fields of the the following free-form filters. The low-dimensional structure of our dynamic parameters makes this regressor more efficient than the regressor for free-form deforma- tion, as it only has three outputs for each full covariance, or even just one for each spherical covariance. Since the covariance is differentiable, the regression is learned end- to-end from the task loss without further supervision.
Figure 6: Gaussian deformation (c-f) structures dynamic receptive fields by controlling the sampling points (blue) according to the covariance. Its low- dimensionality less general but more efficient for learning and inference than free-form deformation (b) by Dai al.) (017)) which requires more parameters.
5 EXPERIMENTS
We experiment with the local recognition task of semantic segmentation, because our method learns local receptive fields. As recognition task, semantic segmentation requires a balance between local scope, to infer where, and global scope, to infer what.
Data CityScapes (Cordts et al., 2016) is a challenging dataset of varied urban scenes from the perspective of a car-mounted camera. We follow the standard training and evaluation protocols and train/validation splits, with 2, 975 finely-annotated training images and 500 validation images. We score results by the common intersection-ovett union metric on the validation set. We evaluate the network itself without post-pres pest-proeessig, test-time augmentation, or other accessorics to isolate the effect of receptive field learning.
Architecture and Optimization For backbones we choose strong fully convolutional networks derived from residual networks (He al.) (016). Dilated residual nets (DRN) (Yu et al. 2017) have high resolution and receptive field size through dilation. Deep layer aggregation (DLA) Yu al. 2018) fuses layers by hierarchical and iterative skip connections. We also define ResNet-34 backbonn as simple alternative for ablations and exploratory experiments, These are representative of common architectural patterns in ofate-oftthe-ar fully convolutional networks.
We train our models by stochastic gradient descent for 240 epochs with momentum 0.9, batch size 16, and weight decay 10-4, Training follows the "poly" learning rate schedule (Chen al. 2018 Zhao et al. 2017) with initial rate 0.01. The imput images are cropped to 800 x 800 and augmented by random scaling, random rotation, and random color distortions as in (Howard) 2013). We train with synchronized, in-place batch normalization (Rota Buld et al. 2018). For fair comparison, we reproduce the DRN and DLA baselines our same setting, which improves on their reported results.
The chosen DRN and DLA architectures are strong methods on their own, but they can be further equipped for learning global spatial transformations and local deformations. Spatial transformer networks (Jaderberg et al.) 2015) and deformable convolution (Dai et al.) 2017) learn dynamic global/local transformations respectively. Spatial transformers serve as a baseline for structure, because they restricted to parametric class of transformations. Deformable convolution serves as a baseline for local, dynamic inference without structure. For comparison in the static setting, we simplify both methods to instead learn static transformations.
Baselines

---

Under review as a conference paper at ICLR 2020
method
I0
DRN-A
72.4
method
10
+ Extra Conv.
72.9
ResNet-34
64.8
method
10
+ STN (static)
70.5
+Blur
66.3
DRN-A
72.4
+ Deformable (static)
72.2
BBlur- Slur-Resaample
68.1
w/ CCL
73.1
+ Composition (ours)
73.5
DoG Blur
70.3
Blur
74.0
DoG Blur-Resee Blur-Resample
71.4
+ CCL
73.1
ASPP
74.1
DRN-A
72.4
+ DoG (ours)
74.1
Blur
74.3
Blur
72.2
DLA-34
76.1
Blur-Resample
73.5
Table Blurring
+ Composition (ours)
78.2
Table 2: It is better to compose with resampling than without.
dilation helps slightly.
Table 1: Learning our composition im-
proves the accuracy of careful designs.
5.1 LEARNING SEMI-STRUCTURED FILTERS
Augmenting Existing Architectures See Table[for the accuracies of the architectures. baselines, and our filtering. We augment the last output stage of DRN-A with a single instance of our compo- sition. We augment the decoder of DLA with instances our composition, one each merge layer. Optimization is end-to-end. Our composition by convolution improves by 1-2 points.
Note that these architectures are already ggressiveey-tuneed and engineering effort. Our composition is still able to deliver improvement through learning without further engineering. the next subsection, we show that joint optimization of our composition assists model search when the chosen architecture is suboptimal.
which required significant model search
How to Compose We can compose with Gaussian structured filter by blurring alone or blurring and resampling. Blurring and resampling first blurs with the Gaussian, and then transforms the sampling points for the following filtering according to the covariance. Blurring and resampling by Gaussian covariance
be interpreted et al. 2015; Yu & Koltun 2016) or as a smooth, affine restriction of deformable convolution (Dai al. 2017). As either be learned end-to-end, we experiment with both in Table 2 From this comparison we choose blurring and resampling for the remainder of our experiments.
smooth, circular extension of dilated convolution (Chen
Blurred Dilation To isolate the effect of blurring without learning, smooth dilation with blur proportional to the dilation rate. CCL (Ding et al. 2018) and ASPP (Chen et al. 2018) are carefully designed dilation architectures for context modeling. but neither blurs before dilating. Improvements from blurred dilation are reported in Table 3 Although the gains are small, this establishes that smoothing can help. This effect should only increase with dilation rate.
5.2 DIFFERENTIABLE RECEPTIVE FIELD SEARCH
Our composition turns choosing receptive fields into task for optimization, instead of design. Table 4 shows how optimization counteracts the reduction of the architectural receptive field size and the enlargement of the input. These controlled experiments, while simple, reflect a realistic lack of knowledge practice: for a new architecture or dataset, the right design unknown.
For these experiments we include our composition in the last stage of the network. We compare fine-tuning only the last stage and end-to-end optimization. End-to-end optimization of our difference of Gaussians significantly reduces the drop in accuracy across scale shifts.
the extreme, we can do structural fine-tuning by including our composition pre-trained network and only optimizing the covariance. When fine-tuning the structure alone, optimization either reduces the Gaussian to delta, doing no harm, or slightly enlarges the receptive field, giving a one point boost. Therefore the special case of the identity, explained in Figure is learnable in practice.
5.3 DYNAMIC GAUSSIAN STRUCTURE
Dynamic inference of the covariance adaptively adjusts receptive fields to vary with the input. In these experiments choose spherical covariance with single degree of freedom for scale.
7

---

Under review as a conference paper at ICLR 2020
Cityscapes Validation
method
params
epoch
IU
A
method
dyn.?
dyn.
10
DRN-A
many
240
72.4
0
params
No Dilation: Smaller Receptive Field
DRN-A
72.4
+ Gauss. Def. (ours)
76.6
ResNet-34
many
240
Free-form Def.
2k
76.6
+ Composition
some
+20
68.1
-4.6
+ DoG
some
+20
68.9
-3.5
ResNet-34
64.8
Bind-to-End
many
240
71.4
-0.8
+ Gauss. Def. (ours)
74.2
Free-form Def.
2k²
75.1
and 2x Enlarged Input
ResNet-34 34
many
240
56.2
-16.2
Cityscapes Test
+ Composition
some
+20
57.8
-14.6
DRN-A
71.2
+ DoG
some
+20
62.7
-9.7
+ Gauss. Def. (ours)
74.3
End-to-Enu
many
240
66.5
-5.9
Free-form Def.
2k
73.6
Table 4: Optimizing receptive fields with our com-
Table 5: Gaussian deformation improves compu-
position helps adjust to the architecture and data.
tational efficiency and rivals free-form accuracy.
(b)
(d)
Figure 7: Qualitative results for dynamic scale inference: (a) inputs, (b) truths. (c) outputs, and (d) scale estimates where small is blue/dark and large is yellow/bright. The scale estimates exhibit structure: coherent segments and boundaries between them can be seen.
Qualitative results for dynamic Gaussian structure are shown Figure The local covariances reflect scale structure in the input and the output segmentation. Quantitative results compare with free-form deformation in Table 5 Gaussian structure improves efficiency while preserving accuracy.
Our results show spherical Gaussian deformation can suffice to achieve equal accuracy general, free-form deformation. Including further degrees of freedom by diagonal and full covariance does not give further improvement this task and data. As scale is ubiquitous transformation in the distribution natural images, dynamic scale inference might suffice to cope with many variations.
CONCLUSION
Composing structured Gaussian and free-form filters makes receptive fields differentiable for direct optimization of the degree of locality. Through receptive field learning, semi-structured parameters do by gradient optimization what current free-form architectures have done by discrete design. That is, in our parameterization changes in structured weights would require changes in free-form architecture.
Our method learns local receptive fields. While we have focused on locality in space, the principle is more general, and extends to locality in time and other dimensions.
Factorization of this sort points to a reconciliation of structure and learning, through which known structure is respected and unknown detail is learned freely.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
J Babaud, A P Witkin, M Baudin, and R O Duda. Uniqueness of the gaussian kernel for scale-space filtering. TPAMI, 1986.
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. TPAMI, 2013.
Burt and E. Adelson. The laplacian pyramid as a compact image code. Communications, IEEE Transactions on, 31 31(4):332-540 1983.
Liang-Chies Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Semantic image segmentation with deep convolutional nets and fully connected CRFs. In ICLR, 2015.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2018.
Taco S Cohen and Max Welling. Steerable cnns. In ICLR, 2017.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Jifeng Dai, Haozhi Qi, Yuwen Xiong. Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
Bert De Brabandere, Xu Jia, Tinne Tuytelaars, and Luc Van Gool. Dynamic filter networks. In NIPS, 2016.
Henghui Ding, Xudong Jiang, Bing Shuai, Ai Qun Liu, and Gang Wang. Context contrasted feature and gated multi-scale aggregation for scene segmentation. In CVPR, 2018.
William T. Freeman and Edward Adelson. The design and use of steerable filters. TPAMI, 1991.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Andrew Howard. Some improvements on deep convolutional neural network based image classifi- cation. arXiv preprint arXiv:1312.5403 2013.
Jorn-Herrik Jacobsen, Jan van Gemert, Zhongyu Lou, and Arnold WM Smeulders. Structured Receptive Fields in CNNs. CVPR, 2016.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukeuoglu. Spatial transformer networks. In NIPS, 2015.
Angjoo Kanazawa, Abhishek Sharma, and David Jacobs. Locally networks. arXiv preprint urXiv: 14/2.5104, 2014.
convolutional neural
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric Xing. Neural architecture search with bayesian optimisation and optimal transport. In NIPS, 2018.
Jan J Koenderink. The structure of images. Biological cybernetics, 50(5):363-370, 1984.
Stephen W Kuffler. Discharge patterns and functional organization of mammalian retina. Journal of neurophysiology, 16(1):37-68, 1953.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Tony Lindeberg. Scale-space theory in computer vision, volume 256. Springer Science & Business Media, 1994.
Tony Lindeberg. Feature detection with automatic scale selection. International joueraaaaanaaouurall of computer vision, 30(2):79-116, 1998.

---

Under review as a conference paper at ICLR 2020
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR, 2019.
D.G. Lowe. Distinctive image features from scale-invariant keypoints. IJV,, 2004.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field deep convolutional neural networks. NIPS, 2016.
Bruno Olshausen, Charles Anderson, and David C Van Essen. neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information. Journal of Neuroscience, 13(1 (((1)):700-4719 1993.
Alan V. Oppenheim and Ronald W. Schafer. Discrete-Time Signal Processing, Prentice Hall Press, Upper Saddle River, NJ, USA, 3rd edition, 2009.
Pietro Perona. Deformable kernels for early vision. TPAMI, 1995.
José Pinheiro and Douglas M Bates. Unconstrained parametrizations for variance-covariance matrices. Statistics and computing, 6(3):289-296, 1996.
Robert W Rodieck and Jonathan Stone. Analysis of receptive fields of cat retinal ganglion cells. Journal of neurophysiology. 28(5)8333889, 1965.
Samuel Rota Bulò, Lorenzo Porzi, and Peter Kontschieder. In-place activated batchnorm for memory- optimized training of dnns. In CVPR, 2018.
Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for semantic segmentation. TPAMI, 2017.
Laurent Sifre and Stéphane Mallat. Rotation, scaling and deformation invariant scattering for texture discrimination. CVPR, 2013.
Eero P Simoncelli and William Freeman. The steerable pyramid: A flexible architecture for multi-scale derivative computation. ICIP, 1995.
Panqu Wang, Pengfei Chen, Ye Yuan. Ding Liu. Zehua Huang. Xiaodi Hou, and Garrison Cottrell. Understanding convolution for semantic segmentation. In WACV, 2018.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016.
Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In CVPR, 2017.
Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In CVPR, 2018.
Richard Zhang. Making convolutional networks shift-invariant again. ICML, 2019.
Rui Zhang, Sheng Tang, Yongdong Zhang, Jintao Li, and Shuicheng Yan. adapivee convolutions for scene parsing. In ICCV, pp. 2031-2039 2017.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
10


Reviews:
[{"review_id": "Skg0uXMXqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a semi-structured composition of free-form filters and structured Gaussian filters to learn the deep representations. Experiments demonstrate its effectiveness in semantic segmentation. The idea is interesting and somewhat reasonable but I still have several concerns. However, I still have several concerns:\n1.\tThe authors proposed to compose the free-form filters and structured filters with a two-step convolution. The authors are expected to clarify why and how the decomposition can realized its purpose? The authors need to further justify the methods by providing more theoretical analysis, and comparing with alternative methods. \n2.\tThe experiments are rather insufficient, and the authors are expected to make more comprehensive evaluations, e.g., more comparisons with the traditional CNN models. \n3.\tThe improvement is rather incremental compared with the alternative methods. The authors actually introduce some prior to the learning process. It would be better if the authors could show some other advantages, e.g., whether it can train the model with smaller number of samples, and whether we can integrate other prior besides Gaussian filters for other structures since Gaussian is a good prior for blurring. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1e3I7w9KH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n- key problem: improved visual representation learning with limited increase in parameters by leveraging Gaussian structure;\n- contributions: 1) compose Gaussian blurs and free-form convolutional filters in an end-to-end differentiable fashion, 2) showing that learning the covariance enables a factorized parameter-efficient representation covering wide and flexibly structured filters, 3) experiments on CityScapes showing the proposed layers can help improve semantic segmentation performance for different architectures (DRN, DLA, and ResNet34).\n\nRecommendation: weak reject\n\nKey reason 1: mismatch between the generality of the claims and experiments.\n- Learning to adapt and optimize receptive fields successfully would be a great fundamental improvement to CNN architectures. Experiments are done on a single dataset for a single task, which seems insufficient to support the generality of the approach and claims in the submission. I would recommend using other datasets (e.g., COCO) and tasks (e.g., object detection, instance segmentation, depth estimation/completion), where the benefits of the approach could be demonstrated more broadly and clearly (including its inherent trade-offs).\n- The improved efficiency (one of the main claims) is only assessed on the number of parameters, which is a direct consequence of the parametrization. Is it significant at the scale of the evaluated architectures? Does it result in runtime performance benefits? If it is indeed a useful structural inductive bias, does it result in improved few-shot generalization performance or less overfitting? Does it enable learning deeper networks on the same amount of data?\n- Why modifying only later layers in the architecture (end of 4.1)? It seems that early layers would make sense too, as it is where most of the downsampling happens.\n\nKey reason 2: lack of clarity and details.\n- Section 1 and the beginning of section 4 are repetitive and verbose; in particular, Sections 4.1 and 4.2 would benefit from less textual descriptions replaced by more concise mathematical formula (simpler in this case), especially in order to know the details behind the methods compared in Tables 1-2-3. \n- Overall, the paper could contain less text describing the hypothetical advantages of the method and the basic preliminaries (section 3), to focus more on the method itself, its details and evaluated benefits. In particular, the dynamic part (section 4.2) is unclear and the method is mostly described in one sentence: \"To infer the local covariances we learn a convolutional regressor, which is simply a convolutional filter.\" Another example of the lack of details is \"many\" vs. \"some\" in the \"params\" column of Table 4.\n- There is also a missed opportunity to provide compelling feature visualizations and qualitative experiments (beyond Fig. 7). For instance, what are the typical covariances learned? What are the failure modes that the proposed modifications address, in particular w.r.t. thin structures and boundaries that are typical hard cases for semantic segmentation and where blurring might be counterproductive?\n\nAdditional Feedback:\n- missing reference: Learning Receptive Field Size by Learning Filter Size, Lee et al, WACV'19;\n- missing reference (w.r.t. local filtering): Pixel-Adaptive Convolutional Neural Networks, Su et al, CVPR'19\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Bye9gaWgqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes semi-structured neural filter composed of structured Gaussian filters and the usual structure-agnostic free-form filters found in neural networks. They are optimized using end-to-end training. Effectively, this lead to increased receptive field size and shape with just few additional parameters. Further, this module is architecture agnostic and can also be integrated with any dynamic inference models. Specifically, when applied on deformable convolutional filters, the deformation at each input can be structured using gaussian filters. Empirical experiments suggest that when integrated with state-of-the-art semantic segmentation architectures, the absolute accuracy on Cityscapes improves by 2%. Large improvement in seen on naive / sub-optimal architectures for segmentation.\n\nGiven that this is first work which demonstrates the efficient composition of classic structured filters with neural layer filters, I believe that research community will benefit to good extent if this paper is accepted.\n\nClarification:\n1. I note that single gaussian is shared across different free-form filters. Is same gaussian also shared across input channels ?\n2. For dynamic inference, what is the sampling resolution used ? How is it related to diagonal elements of covariance ? 2\\sigma ?\n3. In case of blurring and resampling, does the model learn another filter for sampling ? To me, sampling seems similar to dynamic inference operation but with static parameters.\n4. As noted in paper, blurring is fundamental hwen dilating. Does DRN-A and DLA-34 models used for comparison in Table 1 includes blurring prior to dilation ?\n\nAdditional experiment:\n1. Does improved receptive field size and shape also lead to improvement in other downstream tasks such as classification, object detection, depth estimation etc. ?\n2. Table 4 shows that the networks with reduced depth when integrated with composed filters can perform as well as large networks. Does this holds true when extended to above tasks ? \n3. I note that in all the presented results, the composed filters are only included at the last few layers. How the results prunes out when included at the lower as well as at the intermediate layers ? Please include a plot of accuracy vs depth (at which it is included).\n4. I am glad to note that Gaussian deformable models performs as good as free-form deformable models with largely reduced parameters. Can you please add total network parameters comparison in Table 5 ? Further, are these also included only at the top few layers ?\n5. In Table 1, DLA-34 + DoG ?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an interesting idea of inserting Gaussian convolutions into ConvNet in order to increase and to adapt effective receptive fields of network units. The reviewers generally agree that the idea is interesting and that the results on CityScapes are promising. However, it is hard not to agree with Reviewer 3, that validation on a single dataset for a single task is not sufficient. This criticism is unaddressed. 
Decision: Reject