Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation
Authors: Hang Gao
Xizhou Zhu
Stephen Lin
Jifeng Dai
Venue:  (2020)
Keywords: Effective Receptive Fields
Deformation Modeling
Dynamic Inference

Abstract: Convolutional networks are not aware of an object's geometric variations, which leads to inefficient utilization of model and data capacity. To overcome this issue, recent works on deformation modeling seek to spatially reconfigure the data towards a common arrangement such that semantic recognition suffers less from deformation. This is typically done by augmenting static operators with learned free-form sampling grids in the image space, dynamically tuned to the data and task for adapting the receptive field. Yet adapting the receptive field does not quite reach the actual goal -- what really matters to the network is the *effective* receptive field (ERF), which reflects how much each pixel contributes. It is thus natural to design other approaches to adapt the ERF directly during runtime. In this work, we instantiate one possible solution as Deformable Kernels (DKs), a family of novel and generic convolutional operators for handling object deformations by directly adapting the ERF while leaving the receptive field untouched. At the heart of our method is the ability to resample the original kernel space towards recovering the deformation of objects. This approach is justified with theoretical insights that the ERF is strictly determined by data sampling locations and kernel values. We implement DKs as generic drop-in replacements of rigid kernels and conduct a series of empirical studies whose results conform with our theories. Over several tasks and standard base models, our approach compares favorably against prior works that adapt during runtime. In addition, further experiments suggest a working mechanism orthogonal and complementary to previous works.

Published as a conference paper at ICLR 2020
DEFORMABLE KERNELS: ADAPTING EFFECTIVE RE- CEPTIVE FIELDS FOR OBJECT DEFORMATION
Hang Gao Xizhou Zhu Steve Lin Jifeng Dai
UC Berkeley Uniiversity of Science and Technology of China "Microsoft Research Asia hanggêeecs www.eeey..eeeeeeoooooeeeeeeeeeee edu, ezra04080ma (stevelin, ji fiddii wiw.:/iiiaaaaooaooooooooe crosoft.. com com
www.ust...... edu. en
http://people eecs berkeley. edu/ htt.a//..eeeetaate/eeerrroaaaaae dormmbble kerke ls
ABSTRACT
Convolutional networks are not aware of an object's geometric variations, which leads to inefficient utilization of model and data capacity. To overcome this is- sue, recent works on deformation modeling seek to spatially reconfigure the data towards a common arrangement such that semantic recognition suffers less from deformation. This is typically done by augmenting static operators with learned free-form sampling grids in the image space. dynamically tuned the data and task for adapting the receptive field. Yet adapting the receptive field does not quite reach the actual goal - what really matters to the network is the effective receptive field (ERF), which reflects how much each pixel contributes. It is thus natural to design other approaches to adapt the ERF directly during runtime.
In this work, we instantiate one possible solution as Deformable Kernels (DKs), a family of novel and generic convolutional operators for handling object deforma- tions by directly adapting the ERF while leaving the receptive field untouched. At the heart of our method is the ability to resample the original kernel space towards recovering the deformation of objects. This approach is justified with theoretical insights that the ERF is strictly determined by data sampling locations and ker- nel values. We implement DKs generic drop-in replacements of rigid kernels and conduct a series of empirical studies whose results conform with our theories. Over several tasks and standard base models, our approach compares favorably against prior works that adapt during runtime. In addition, further experiments suggest working mechanism orthogonal and complementary to previous works.
INTRODUCTION
The rich diversity of object appearance in images arises from variations in object semantics and deformation. Semantics describe the high-level abstraction of what we perceive, and deformation defines the geometric transformation tied to specific data (Gibson 1950). Humans are remarkably adept at making abstractions of the world (Hudson & Manning 2019); we see in raw visual signals, abstract semanties away from deformation, and form concepts.
Interestingly. modern convolutional networks follow an analogous process by making abstractions through local connectivity and weight sharing (Zhang 2019). However, such mechanism is an inefficient one, as the emergent representations encode semantics and deformation together, instead of as disjoint notions. Though a convolution responds accordingly to each input, how it responds is primarily programmed by its rigid kernels, as in Figure [(a, b) In effect, this consumes large model capacity and data modes (Shelhamer et al. 2019).
We argue that the awareness of deformations emerges from adaptivity the ability to adapt at run- time (Kanazawa et al., 2016 Jia et al., 2016 Li et al. 2019). Modeling of geometric transformations has been constant pursuit for vision researchers over decades (Lowe et al., 1999; Lazebnik et al. 2006 Jaderberg et al. 2015) Dai et al., 2017). A basic idea is to spatially recompose data towards common mode such that semantic recognition suffers less from deformation. A recent work that
"Equal contributions. Work is done when Hang and Xizhou are interns Microsoft Research Asia.

---

Published as a conference paper at ICLR 2020
scaling: 1.25x
+
rotation: 30
(a) rigid kern. (norm)
(b) rigid kern. (deform)
(c) deformable conv.
(d) deformable kern.
Figure 1: Adaptation for deformation. We show how different 3 x 3 convolutions interact with deformations two images. Kernel spaces are visualized as flat 2D Gaussians. Each "+" indicates computation between pixel and kernel value sampled from the data and kernel space. Their colors represent corresponding kernel values. (a, b) Rigid kernels cannot adapt to specific deformations, thus consuming large model and data capacity. (c) Deformable Convolutions (Dai al. 2017) reconfigure data towards common arrangement to counter the effects of geometric deformation. (d) Our Deformable Kernels (DKs) instead resample kernels and, in effect, adapt kernel spaces while leaving the data untouched. Note that (b) and (c) share kernel values but sample different data locations, while (b) and (d) share data locations but sample different kernel values.
is representative of this direction is Deformable Convolution (Dai et al. 2017 Zhu et al. (019). As shown in Figure [[c]] it augments the convolutions with free-form sampling grids the data space. It is previously justified adapting receptive field, or what we phrase as the "theoretical receptive field", that defines which input pixels can contribute to the final output. However, theoretical re- ceptive field does measure how much impact input pixel actually has. On the other hand, Luo et al. (2016) propose to measure the effective receptive field (ERF), i.c. the partial derivative of the output with respect the input data, to quantify the exact contribution of each pixel the convolution. Since adapting the theoretical receptive field is not the goal but a means to adapt the ERF, why not directly tune the ERF specific data and tasks runtime?
Toward this end, we introduce Deformable Kernels (DKs), family of novel and generic convolu- tional operators for deformation modeling. We aim augment rigid kernels with the expressiveness to directly interact with the ERF of the computation during inference. Illustrated in Figure [(d] DKs learn free-form offsets on kernel coordinates deform the original kernel space towards specific data modality, rather than recomposing data. This can directly adapt ERF while leaving receptive field untouched. The design of DKs that agnostic data coordinates naturally leads two vari- ants the global DK and the local DK, which behave differently in practice we later investigate. We justify our approach with theoretical results which show that ERF is strictly determined by data sampling locations and kernel values. Used as a generic drop-in replacement of rigid kernels, DKs achieve empirical results coherent with our developed theory. Coneretely, we evaluate our operator with standard base models image classification and object detection. DKs perform favorably against prior works that adapt during runtime. With both quantitative and qualitative analysis, we further show that DKs can work orthogonally and complementarily with previous techniques.
2 RELATED WORKS
We distinguish our work within the context of deformation modeling as our goal, and dynamic inference our means.
2

---

Published as a conference paper at ICLR 2020
We refer deformation modeling learning geometric transformations in 2D image space without regard to 3D. One angle to attack deformation modeling is to craft cer- tain geometric invariances into networks. However, this usually requires designs specific to certain kinds of deformation, such as shift, rotation, reflection and scaling (Sifre & Mallat 2013 Bruna & Mallat 2013: Kanazawa al., 2016] Cohen & Welling 2016 Worrall et al.] 2017 Esteves al.; 2018). Another line of work on this topic learns to recompose data by either semi-parameterized or completely free-form sampling image space: Spatial Transformers (Jaderberg al. 2015) learns 2D affine transformations, Deep Geometric Matchers (Rocco et al., 2017) learns thin thin-plate spline transformations, Deformable Convolutions (Dai al. 2017 Zhu et al. 2019) learns free-form trans- formations.
Deformation Modeling:
We interpret sampling data space as an effective approach to adapt effective receptive fields (ERF) by directly changing receptive field. At high-level, our Deformable Kernels (DKs) share intu- itions with this line of works for learning geometric transformations, yet are instantiated by learning to sample in kernel space which directly adapt ERF while leaving theoretical receptive fields un- touched. While kernel space sampling is also studied in Deformable Filter (Xiong et al. (011)) and KPConv (Thomas et al. 2019), but in their contexts, sampling grids are computed from imput point clouds rather than learned from data corpora.
Dynamic Inference:
Dynamic inference adapts the model individual operators to the observed data. The computation our approach differs from self-attention (Vaswani et al. 2017 Wang et al. 2018) in which linear or convolution modules augmented with subsequent queries that extract from the same input. We consider our closest related works in terms of implementation as those approaches that adapt convolutional kernels run time, It includes but is not limited Dynamic Filters (Jia et al.) 2016), Selective Kernels (Li et al. 2019) and Conditional Convolutions (Yang al., 2019). All of these approaches can learn and infer customized kernel spaces with respect to the data, but are either less inefficient or are loosely formulated. Dynamic Filters generate new filters from scratch, while Conditional Convolutions extend this idea to linear combinations of synthesized filters. Selective Kernels are, on the other hand, comparably lightweight, but aggre- gating activations from kernels of different size is not as compact as directly sampling the original kernel space. Another line of works contemporary to ours (Shelhamer et al. 2019: Wang et al., 2019) is to compose free-form filters with structured Gaussian filters, which essentially transforms kernel spaces by data. Our DKs also differ from these works with the emphasize of direct adaptation the ERF rather than the theoretical receptive field, As mentioned previously, the true goal should be adapt the ERF, and knowledge, our work the first study dynamic inference of ERFs.
set
APPROACH
We start by covering preliminaries on convolutions, including the definition of effective receptive field (ERF). We then formulate theoretical framework for analyzing ERFs, and thus motivate our Deformable Kernels (DKs). We finally elaborate different DK variants within such a framework. Our analysis suggests compatibility between DKs and the prior work.
3.1 A DIVE INTO CONVOLUTIONS
2D Convolution:
Let us first consider an input image I E DDD. By convolving it with ker- nel W E R of stride 1, we have an output image O whose pixel values each coordi-
nate E R² can be expressed as
O O, >I=\
REK
by enumerating discrete kernel positions & within the support K. K= rigid grid for sampling data and kernels.
nz. This defines
Theoretical Receptive Field: The same kernel W can be stacked repeatedly to form linear con- volutional network with layers. The theoretical receptive field can then be imagined as the "ac- cumulative coverage" of kernels at each given output unit on the input image by deconvolving back through the network. This property characterizes set of input fields that could fire percepts onto corresponding output pixels. The size of a theoretical receptive field scales linearly with respect to the network depth and kernel size K (He et al. 2016).
3

---

Published as a conference paper at ICLR 2020
Effective Receptive Field: Intuitively. not all pixels within theoretical receptive field contribute equally. The influence of different fields varies from region to region thanks to the central emphasis of stacked convolutions and also the non non-linearity induced by activations. The notion of effective receptive field (ERF) (Luo et al. 2016) thus introduced to measure the impact of each input pixel on the output given locations. It is defined partial derivative field of the output with respect to the input data. With the numerical approximations in linear convolution networks, the ERF was previously identified as Gaussian-like soft attention map over input images whose size grows fractionally with respect to the network depth n and linearly to the kernel size K. Empirical results validate this idea under more complex and realistic cases when networks exploit non-linearities, striding, padding, skip connections, and subsampling.
3.2 ANALYSIS ON EFFECTIVE RECEPTIVE FIELDS
We aim to revisit and complement the previous analysis on ERFs by Luo et al. (2016). While the previous analysis concentrates studying the expectation of an ERF, 1.e., when network depth approaches infinity or all kernels are randomly distributed without learning in general, our analysis focuses how can perturb the computation such that the change in ERF is predictable, given an input and a set of kernel spaces.
We start analysis by considering linear convolutional network, without any unit activations, as defined in Section 3.1 For consistency, superscripts are introduced to image I, kerne W, and subscripts kernel positions & to denote the index [], of each layer. Formally, given an input image ((0) and set of K K kernels (W()))n )== of stride 1, we can roll out the final output O [(n) by unfolding Equation 1
Σ
jthn wr())
(m-1)((m))
Σ
(n-2) tt
W (m) (=(n1] [n-1]
An ex
(Kn An)E?
Σ
(0) 3+22
((()
h
By definition the effective receptive field value R (i:J) = 0r /oI! of output coordinate that takes input coordinate i can be computed by
R (t:5)
Σ
1 Σ k. i
a
(le Ag........... JEK""
where [[.) denotes the indicator function. This result indicates that ERF is related only to the data sampling location j. kernel sampling location k. and kernel matrices (W(*)).
If we replace the m kernel (mm) with 1 1 kernel of single parameter W (m) it, the value of ERF becomes to
sampled from
km
(n) km))
Σ
11j+
k. =
W
w/(m)) (4)
km
(k An- Am.ty....................k. A.)en-- ".)
SE5
S
where S = [1, n] {m]. Since a K x K kernel can be deemed composition of K2 1x kemels distributed on a square grid, Equation can thus be reformulated as
R. (())
Σ (( (i:
km: kmn).
Am EK
For the case of complex non-lincarities, where we here consider post ReLt activations in Equa- tion 1
O, max Σ I4
(6)
lEK
The original definition of ERF in Luo al. 2016 (0, 0), to partially avoid the effects zero padding. this work, we will keep j favor generality while explicitly assuming input size D + oc.
focuses the central coordinate of the output, i.e.
Our analysis currently only considers the ReL.U network for its nice properties and prevalent popularity.
4

---

Published as a conference paper at ICLR 2020
We can follow similar analysis and derive corresponding ERF
R (m) ( km)=
Σ
cl (1: ki, ki,
kas km)
II
W
Wm
km
(11 Mm 1: 1 kn))km=i
#ES
where c (f: k Ans km)) = 1 [ + Σ kg =
>
W > 0].
h
Rm
Here can that the ERF becomes data-dependent due to the coefficient C, which is tied to input coordinates, kernel sampling locations, and input data [(0], The more detailed analysis of this coefficient is beyond the scope of this paper. However, it should
noted that this coefficient only "gates" the contribution of the input pixels to the output. So in practice, ERF is "porous" there inactive (or gated) pixel units irregularly distributed around the ones that fire. This phenomenon also appeared in previous studies (such Luo et al. (2016), Figure 1). The maximal size of an ERF is still controlled by the data sampling location and kernel values as in the linear cases in Equation 5
A nice property of Equation and Equation 5 is that all computations are linear, making it com- patible with any linear sampling operators for querying kernel values of fractional coordinates. In other words, sampling kernels in effect samples the ERF on the data in the linear case. but also roughly generalizes to non-linear cases as well. This finding motivates our design of Deformable Kernels (DKs) in Section 3.3
3.3 DEFORMABLE KERNELS
In the context of Equation resample the kernel W with group of learned kernel offsets denoted as {Ak} that correspond to each discrete kernel position k. This defines our DK as
O2
Σ KEK
14th Wk+ak:
and the value of ERF as
R (t:5)
Σ EKC
R( (t:j + km km + Akm)
(8)
Note that this operation leads sub-pixel sampling in the kernel space. In practice, we use bilinear sampling to interpolate within the discrete kernel grid.
Intuitively, the size (resolution) of the original kernel space can affect sampling performance. Con- cretely. suppose we want to sample 3 x 3 kernel. DKs do not have any constraint on the size of the original kernel space, which we call the "scope size" of DKs. That said, we can use a W of any size K' even though the number of sampling locations is fixed as K We can thus exploit large kernels- - the largest ones can reach 9 x in our experiments with nearly no overhead in computation since bilinear interpolations can also increase the number of learning parameters, which in practice might become intractable not handled properly. In our implementation, will exploit depthwise convolutions (Howard al., 2017) such that increasing scope size induces negligible amount of extra parameters.
extremely lightweight compared to the cost of convolutions. This
As previously discussed, sampling the kernel space effect transforms into sampling the ERF. On the design of locality and spatial granularity of our learned offisets, DK naturally delivers two variants the global DK and the local DKs, as illustrated in Figure In both operators, we learn kernel offset generator G that maps an input patch into set of kernel offsets that are later applied to rigid kernels.
In practice, we implement Gelabal as a stack of one global average pooling layer, which reduces feature maps into a vector, and another fully-connected layer without non-linnarisies, which projects the reduced vector into an offset vector of 2 dimensions. Then, we apply these offsets to all convolutions for the input image following Equation 7 For local DKs, we implement Glccal as an extra convolution that has the same configuration the target kernel, except that it only has 2K² output channels. This produces kernel sampling offsets {Ak] that are additionally indexed by output locations j. It should be noted that similar designs were also discussed in Jia et al. (2016), in which filters are generated given either an image or individual patches from scratch rather than by resampling.
5

---

Published as a conference paper at ICLR 2020
local deformable kerm. global deformable
Golatall
gltsbal
Eiiiall AngPoal
FC
Gacal
nocal
Giocul
Conv
Figure 2: Instantiations. We show how DK variants works with an example image that contains a large and small object. (a) The global DK learns one set kernel sampling grid given an input image and apply it to all data positions. (b) The local DK adapts kernels for each input patches, and induces better locality for deformation modeling.
Intuitively, we expect the global DK to adapt kernel space between different images but not within a single input. The local DK can further adpat to specific image patches: for smaller objects, is better to have shaper kernels and thus denser ERF; for larger objects, flatter kernels can be more beneficial for accumulating a wider ERF. On a high level, local DKs can preserve better locality and have larger freedom to adapt kernel spaces comparing to its global counterpart. We later compare these operators in our experiments.
3.4 LINK WITH DEFORMABLE CONVOLUTIONS
The core idea of DKs is learn adaptive offsets to sample the kernel space for modeling deforma- tion, which makes them similar to Deformable Convolutions (Dai et al. 2017] Zhu et al., 2019), at both the conceptual and implementation levels. Here, distinguish DKs from Deformable Convo- lutions and show how they can be unified.
Deformable Convolutions be reformulated in general form
O, Σ Is-Kーaş Wk. REK
where they aim learn group of data offsets {Aj} with respect discrete data positions j. For consistency for analysis, the value of effective receptive field becomes
R ({:
Σ R( (8:j km + Ajm, km).
(10)
Ry EK
This approach essentially recomposes the input image towards common modes such that semantic recognition suffers less from deformation. Moreover, according to our previous analysis Equa- tion sampling data is another way of sampling the ERF. This, to certain extent, also explains why Deformable Convolutions are well suited for learning deformation-agnoos representations.
Moreover, learn both data and kernel offsets in convolutional operator. Conceptually, this can be done by merging Equation 7 with Equation 9 which leads to
1y4k+A WaTAK.
KEK
(11)
R (n) DEADR (i:j)=
Σ
R()) EK
km Ajm, km + Akm).
We also investigate this operator in our experiments. Although the two techniques may be viewed as serving similar purpose, we find the collaboration between Deformable Kernels and Deformable Convolutions be powerful in practice, suggesting strong compatibility.

---

Published as a conference paper at ICLR 2020
4 EXPERIMENTS
We evaluate our Deformable Kernels (DKs) image classification using ILSVRC and object detec- tion using the COCO benchmark. Necessary details are provided to reproduce our results, together with descriptions base models and strong baselines for all experiments and ablations. For task- specific considerations, we refer to each corresponding section.
Implementation Details: We implement operators in PyTorch and CUDA. We exploit depth- wise convolutions when designing our operator for better computational efficiency We initialize kernel grids to be uniformly distributed within the scope size. For the kernel offset generator, its learning rate to be fraction of that of the main network, which we cross-validate for each base model. We also find it important to clip sampling locations inside the original kernel space, such that & + Ak E IC in Equation]] 7
set
We choose our base models be ResNet-50 (He et al. 2016) and MobileNet- V2 (Sandler et al. 2018), following the standard practice for most vision applications. As men- tioned. we exploit depthwise convolution and thus make changes the ResNet model. Concretely, we define our ResNet-50-DW base model by replacing all 3 x 3 convolutions by its depthwise coun- terpart while doubling the dimension of intermediate channels in all residual blocks. We find to be reasonable base model compared to the original ResNet-50, with comparable performance on both tasks. During training, we set the weight decay to be 4 x 10-5 rather than the common 10-4 for both models since depthwise models usually underfit rather than overfit (Xie et al., 2017] Howard ctt al. 2017 Hu et al. 2018). We set the learning rate multiplier of DK operators as 10 for ResNet-50-DW and 10 for MobileNer-V2 in all of our experiments.
Base Models:
Strong Baselines: We develop our comparison with two previous works: Conditional Convolu- tions Yang et al. 2019) for dynamies inference, and Deformable Convolutions (Dai et al. 2017 Zhu et al. 2019) for deformation modeling. We choose Conditional Convolutions due to similar computation forms sampling can be deemed as an elementewise "expert voting" mechanism. For fair comparisons, We reimplement and reproduce their results. We also combine our operator with these previous approach show both quantitative evidence and qualitative insight that our working mechanisms are compatible.
4.1 IMAGE CLASSIFICATION
We first train our networks on the ImageNet 2012 training set (Deng et al. 2009). We adopt common experiment protocol for fair comparisons as in Goyal et al. (2017; ; Loshchilov & Hutter
(0177) For more details, please refer our supplement.
We first ablate the scope size of kernels for our DKs and study how it can affect model performance using ResNet-50-DWi As shown in Table[]] our DKs are sensitive to the choice of the scope size. We shown that when only applied to 3 x 3 convolutions inside residual bottlenecks, local DKs induce +0.7 performance gain within the original scope. By further enlarging the scope size, performance increases yet quickly plateaus at scope x 4, yielding largest +1.4 gain for top-1 accuracy. Our speculation is that, although increasing scope size theoretically means better interpolation, it also makes the optimization space exponentially larger for each convolutional layer. And since number of entries for updating is fixed, this also leads to relatively sparse gradient flows. In principle, we set default scope size at 4 x for our DKs.
We next move on and ablate our designs by comparing the global DK with the local DK, shown in the table. Both operators helps while the local variants consistently performs better than their global counterparts, bringing a +0.5 gap on both base models. We also study the effect of using more DKs in the models the x 1 convolutions are replaced by global D with scope 2 x 2. Note that all x 1 convolutions are not depthwise, and therefore this operation induces nearly 4 times of parameters. We refer their results only for ablation and show that adding more DKs still helps especially for MobileNet-V V2 since it is under- parameterized. This finding also holds for previous models Yang et al. 2019) well.
This makes enlarging the kernel scope size tractable and prevents extensive resource competition CUDA kernels when applying local DKs. The implementation of local DKs right now cannot support large number of output channels.
7

---

Published as a conference paper at ICLR 2020
Backbone
1xl Deformable Kernels
3x3 Deformable Kernels
topl (%)
#P (M)
GFLOPs
ResNet-50
w/o
w/o
76.7
25.6
3,86
Res
w/o
w/o
76.3
23.7
3.82
local. scope size 3x3
77.4
24.9
4.32
local, scope size 4x4
78.1
25.0
4.32
ResNet-50-DW
w/o
local, scope size 5x5
77.8
25.0
4.32
local, scope size 9x9
77.4
25.4
4.32
w/o
global, scope 4x
77.6
23.9
3.82
globul, scope size 2x2
global, scope size 4x4
77.9
80.1
4.09
ResNet-50-DW
W/o
local, scope size 4
78.1
25.0
4.32
global, scope size 2x2
local, scope size w/o
78.5
81.2
4.60
MobileNet-V2
w/o
71.9
3.5
0.31
w/o
global, scope size 4x4
73.6
3.7
0.31
global, scope size 2x2
global, scope size 4x4
74.5
10.1
0.34
MobileNet-V2
w/o
local, scope size 4x4
74.1
4.7
0.73
global, scope size 2x2
local, scope size 4x
74.8
11.1
0.76
Table 1:
Ablations of scope size and different instantiations of DK for image classification.
Using proper scope size, and more DK layers boosts performance. Modeling individual offset kernel grid for each data entries is also beneficial.
Backbone
x Deformable Kernels
3x3 Deformable Kernels
topl (%)
(M)
GFLOPs
ResNet-50-DW
w/o
local, scope size 4x4
78.1
25.0
4.32
ResNet-50-DW
w/o local, scope size 4x4
77.6
42.5
7.13
w/o
with SCC
78.9
43.7
7.61
ResNet-50-DDW
w/o
78.0
24.8
4.10
w/o
with DCN
local, scope size 4x4
79.0
26.1
4.60
MobileNet-V2
w/o
local, scope size 4x4
74.1
4.7
0.73
MobileNet-V2
w/o
74.3
19.0
2.19
w/o
with SCC
local, scope size 4x
75.5
19.7
2.48
MobileNet-V2
w/o
73.2
4.6
0.52
w/o
with DCN
local, scope size 4
74.4
5.8
0.93
Table 2: Comparisons to strong baselines for image classification DKs perform comparably or superiorly previous methods. Further combinations yield consistent gain, suggesting orthogonal and compatible working mechanisms.
We further compare and combine DKs with Conditional Convolutions and Deformable Convolu- tions. Results are recorded in Table We can see that DKs perform comparably on ResNet- V2 and compare favorably MobileNet-V2 improve +0.9 from Deformable Convolutions and achieve comparable results with less than a quarter number of parameters compared to Conditional Convo- lutions. Remarkably, in reach. We see consistent boost in top-1 accuracy compared to strong baselines: +1.3/+1.0 on ResNet-50-DW, and 11.2/+1.2 MobileNet-V2. These gaps are bigger than those from ablation, suggesting the working mechanisms across the operators to be orthogonal and compatible.
also show that if combined together,
larger performance gains
4.2 OBJECT DETECTION
We examine DKs on the COCO benchmark (Lin al. 2014). For all experiments, we use Faster R-CNN (Ren et al. 2015) with FPN (L.in et 2017) as the base detector, plugging in the backbones we previously trained on ImageNet. For MobileNet-V V2, we last feature maps of the each resolution for FPN post aggregation. Following the standard protocol, training and evaluation the 120k images the train-vll split and the 20k images in the test dev split, respectively. measure the standard mean average precision (mAP) and shattered scores for small, medium and large objects.
performed on
For evaluation,
Table 3 and Table the same style of analysis in image classification. While the baseline methods of ResNet achieve 36.6 mAP, indicating a strong baseline detector, applying local DKs brings +1.2 mAP improvement when replacing 3x3 rigid kernels alone and a 11.8 mAP improve-
8

---

Published as a conference paper at ICLR 2020
Backbone
1x Deformable Kernels
3x3 Deformable Kemels
mAP
mAPs
mAPM
mAPL
ResNet-50-DW
w/o
w/o
36.6
22.1
39.9
46.6
w/o
global, scope 4x4
36.7
22.6
40.2
46.9
global, scope size 2x2
global. scope size 4x4
37.1
23.1
ResNet-50-DW
w/o
local, scope size 4x4
37.8
23.4
41.6
48.2
global, scope size 2x2
local, scope size 4x w/o
38.4
23.4
42.0
49.4
MobileNct-V2
w/o
31.3
18.6
33.7
40.4
w/o
global, scope size 4x4
32.5
19.6
35.8
41.9
global, scope size 2x2 w/o
global, scope size 4x4
32.9
19.4
35.5
42.8
MobileNet-V2
local, scope
32.9
42.5
global, scope size 2 2x2
local, scope size 4x
33.7
20.2
36.7
44.0
Table 3: Ablations for object detection.
Consistent results with image classification.
Backbone
1x Deformable Kernels
3x3 Deformable Kemels
mAP
mAPs
mAPM
mAP:
ResNet-50-DDV
global, scope size 2x2
local, scope size 4x4
38.4
23.4
42.0
49.4
ResNet-50-DW
36.3
22.1
39.3
47.0
w/o w/o
with SCC
local, scope size x4
38.0
23.4
41.9
48.4
ResNet-50-DW
w/o
39,9
24.0
43.4
52.6
with DCN
local, scope size
40.6
24.6
53.3
MobileNet-W V2
global, scope size 2x2
local, scope size 4
33.7
20.2
36.7
44.0
MobileNet-N V2
w/o
33.2
20.5
35.6
43.3
w/o
with SCC
local, scope size 4x4
34.3
20.2
37.3
MobileNet-V V2 with DCN
w/o
34.4
20.5
37.0
44.7
w/o
local, scope size 4x4
35.6
20.6
38.5
47.3
Table 4: Comparisons to strong baselines for object detection DKs perform fall short to De- formable Convolution, but combination still improves performance.
ment when replacing both 1x1 and 3x3 rigid kernels. This trend magnifies on MobileNet-v2 models, where see improvement of +1.6 mAP and +2.4 mAP, respectively. Results also confirm the effectiveness of local DKs against global DKs, which is again in line with our expectation that local DKs can model locality better.
For the comparisons with strong baselines, an interesting phenomenon worth noting is that though DKs perform better than Deformable Convolutions on image classification, they fall noticeably short for object detection measured by mAP. We speculate that even though both techniques can adapt ERF in theory (as justified Section 3.2). directly shifting sampling locations on data easier to optimize. Yet after combining DKs with previous approaches, we can consistently boost per- formance for all the methods- +0.7/+1.2 for Deformable Convolutions on each base models, and +1.7/+1.1 for Conditional Convolutions. These findings align with the results from image classifi- cation. next investigate what DKs learn and why they are compatible with previous methods in general.
4.3 WHAT DO DEFORMABLE KERNELS LEARN?
Awareness of Object Scale: Since deformation is hard to quantify, we use object scale as a rough proxy to understand what DKs learn. In Figure 3 we show the t-SNE (Maaten & Hinton 2008) of learned model dynamics by the last convolutional layers in MobileNet V2 using Conditional Convolution and our DKs. We validate the finding as claimed by Yang et al. (2019) that the experts of Conditional Convolutions have better correlation with object semantics than their scales (in reference to Figure 6 from their paper). Instead, our DKs learn kernel sampling offsets that strongly correlate to scales rather than semantics. This sheds light why the two operators complementary in our previous experiments.
Adaptation of Effective Receptive Fields: practice, we show ERF visualizations on a set of images in which they display different degrees of deformations. We compare the results of rigid kernels, Deformable Convolutions, combination of the two operators. For all examples, note that the theoretical receptive field covers every pixel in the image but ERFs contain only central portion of it. Deformable Convolutions
To verify our claim that DK indeed adapts ERFs in
DKs, and the

---

Published as a conference paper at ICLR 2020
semantics
scales
semantics.
scales
(a) conditional conv. expert scores (8)
(b) deformable kern. kernel offsets (18-2x3k3)
Figure 3: Semanties vs. Scales. We show t-SNE results learned model dynamics using 10 ran- dom classes of objects from the COCO test- -ev split. Each point represents an object extracted by ground-truth bounding box, whose color either denotes its class label or bounding box scale. The color of an object scale is its normalized area rank discretized by every 10th percentile among all data. Numbers inside parentheses indicate the dimension of learned dynamics before t-SNE. (a) The dynamics of Conditional Convolutions are closer to semantics than to object scales. (b) On the contrary, our DKs learn dynamics that are significantly related to scales rather than semantics.
FONDAN
person
person
(a) rigid kem.
(b) deformable conv.
(c) deformable kern.
(d) deformable conv. kern.
Figure 4: Learned Effective Receptive Fields. We show learned ERFs three images with large, medium, and small objects from the COCO test-dev split. Given each ground-truth bounding box, visualize the non-zero ERF values of its central point. Theoretical RFs cover the whole image for all three examples and we thus ignore them in our plots. (a) Rigid kernels have strong central effects and Gaussian-like ERF that cannot deal with object deformation alone. (b) De- formable Convolutions and (c) Deformable Kernels both tune ERFs to data. (d) Combining both operators together enables better modeling 2D geometric transformation of objects.
and DKs perform similarly in terms of adapting ERFs, but Deformable Convolutions tend to spread out and have sparse responses while DKs tend to concentrate and densely activate within an object region. Combining both operators yields more consistent HRFs that exploit both of their merits.
CONCLUSION
In this paper, we introduced Deformable Kernels (DKs) to adapt effective receptive fields (ERFs) of convolutional networks for object deformation. We proposed sample kernel values from the original kernel space. This in effect samples the ERF in linear networks and also roughly gener- alizes to non-linear cases. We instantiated two variants of DKs and validate our designs. showing connections to previous works. Consistent improvements over them and compatibility with them were found, as illustrated in visualizations.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. 7PAMI, 2013.
Taco Cohen and Max Welling, Group equivariant convolutional networks. In ICML, 2016.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In CVPR, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
Carlos Esteves, Christine Mieen-Blacchete,, Xiaowei Zhou, and Kostas Danillidis. Polar transformer networks. In ICLR, 2018.
James Gibson. The perception of the visual world. Houghton Mifflin, 1950.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:
2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. In CVPR, 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exciiatton networks. In CVPR, 2018.
Drew A Hudson and Christopher Manning. Learning by abstraction: The neural state machine. arXiv preprint arXiv:1907.03950, 2019.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, al. Spatial transformer networks. In
NewrIPS, 2015.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks. In NeurlPS, 2016.
Angjoo Kanazawa, Abhishek Sharma, and David W. Jacobs. Locally ceale-invariant convolutional neural networks. In NeurIPS Workshop, 2016.
Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In CVPR, 2006.
Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. CVPR, 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
David G Lowe al. Object recognition from local scale-invariant features. ICCV, 1999.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. In NewrIPS, 2016.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.

---

Published as a conference paper at ICLR 2020
Ignacio Rocco, Relja Arandjelovic, and Josef Sivic. Convolutional neural network architecture for geometric matching. In CVPR, 2017.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenccks. In CVPR, 2018.
Evan Shelhamer, Dequan Wang, and Trevor Darrell. Blurring the line between structure and learning to optimize and adapt receptive fields. arXiv preprint arXiv: 1904. 1487, 2019.
Laurent Sifrr and Stéphane Mallat. Rotation, scaling and deformation invariant scattering for texture discrimination. CVPR, 2013.
Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In ICCV, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Dequan Wang, Evan Shelhamer, Bruno Olshausen, and Trevor Darrell. Dynamic scale inference by entropy optimization. arXiv preprint (00.:::99900311/6 2019.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov. and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In CVPR, 2017.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR. 2017.
Yuwen Xiong, Mengye Ren, Renjie Liao, Kelvin Wong, and Raquel Urtasun. Deformable filter convolution for point cloud reasoning. arXiv preprint arXiv: 13079, 2019.
Brandon Yang, Gabriel Bender, Quoc V Le, and Jiquan Ngiam. Soft conditional computation. arXiv preprint arXiv:1904 04971, 2019.
Richard Zhang. Making convolutional networks shift-invariant again. ICMI., 2019.
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In CVPR, 2019.
12

---

Published as a conference paper at ICLR 2020
image patch
kernel patch
W
B
{AA]
W
*
Illustration of forarddg through 3x3 local Deformable Kernel from 4x4 scope. For each input patch, local DK first generates group of kernel offsets {åk} from input feature patch using the light-weight generator G (a 3 x convolution of rigid kernel). Given the original kernel weights W and the offset group { {āk}, DK samples new set of kernel W using bilinear sampler B. Finally, DK convolves the imput feature map and the sampled kernels complete the whole computation.
Figure 5:
A COMPUTATION FLOW OF DEFORMABLE KERNELS
We now cover more details on implementing DKs by elaborating the computation flow of their forward and backward passes. We will focus on the local DK given its superior performance in practice. The extension global DK implementation is straight-forward
A.1 FORWARD PASS
we introduce kernel offset generator G and bilinear sampler B. Figure illustrates an example of the forward pass.
In Section 3.3
Concretely, given a kernel W and learned group of kernel offsets {Ak} top of regular 2D grid {k), we can resample a new kernel W by a bilinear operator B as
W' Wk-AA -A = Σ
B(k + Ak. W'r
(12)
WEEK
where Biks Ak, k') max
|ki + Ak= - K2 max(0, [Ky Aki K,D)
Given this resampled kernel, DK convolves it with the input image just as normal convolutions using rigid kernels, characterized by Equation] [
A.2 BACKWARD PASS
The backward pass of local DK consists of three types of gradients: (1) the gradient to the data of the previous layer, (2) the gradient the full scope kernel of the current layer and (3) the additional gradient to the kernel offset generator of the current layer. The first two types of gradients share same forms of the computation comparing the normal convolutions. We now cover the computation for the third flow of gradient that directs where to sample kernel values.
In the context of Equation the partial derivative of a output item Og w.r.t. x component of given kernel offset Ak, (similar for its y component Akyy) can be computed as
00, El1t Σ
aB + Ak, k')
WR
(13)
dak.
OAkr
0
kkg + Ah W/ > 2
0B(k + Ak, k')
where JAkr
max (0, iky Akin K2l)
1
h Akir R2
h + kis 2 ki
13

---

Published as a conference paper at ICLR 2020
Output 112 x 112
ResNet-50
ResNet-50-DW
7 x 7, 64, stride 2
56 x 56
3 X 3 max pool, stride 2
X 64
1 x 1, 128
56 x 56
3 x 3, 64
x
x 3, 128, G = 128
x 3
x 1, 256
x 1, 256
X 1, 128
x 1, 256
28 x 28
3 x 3, 128
X
X 3, 256, G = 256
x
1 512
X 512
1 x 256
1 X 512
14 x 14
3, 256
x 6
x 3, 512, G 512
x 6
X 1, 1024
1 1, 1024
512
x 1, 1024
7
312 512
3 3, 1024, =0024
3
1, 2048
x 2048
1 #P (M)
7 global average pool, 1000-d fc, softmax
25.6
23.7
GFLOPss
3.86
3.82
Table 5: Network architecture of our ResNet-50-DW comparing to the original ResNet-50 Inside the brackets are the general shape residual block, including filter sizes and feature di- mensionalities. The number of stacked blocks on each stage is presented outside the brackets. "G = 128" suggests the depthwise convolution with 128 input channels. Two models have sim- ilar numbers of parameters and FLOPs. At the same time, depthwise convolutions facilitate the computation efficiency of our Deformable Kernels.
B NETWORK ARCHITECTURES
Table shows the comparison between the original ResNet-50 (He al. 2016) and our modified ResNet-50-DW. The motivation of introducing depthwise convolutions to ResNet is to accelerate the computation of local DKs based on our current implementations. The ResNet-50-DDe model has similar model capacity/complexity and performance (see Table [ compared to its non-depthwise counterpart, making it an ideal base architecture for our experiments.
On the other hand, in all of our experiments, MobileNet-Ve V2 (Sandler et al. 2018 base model is left untouched.
C ADDITIONAL COMPARISON OF EFFECTIVE RECEPTIVE FIELDS
We here show additional comparison ERFs when objects have different kinds of deformations in Figure 6 Comparing to baseline, our method can adapt ERFs to be more persistent to object's semantic rather than its geometric configuration.
D ADDITIONAL. EXPERIMENT DETAILS
Image Classification: Similar to Goyal et al. (011)): Loshchilov Hutter (2017), training is per- formed by SGD for 90 epochs with momentum 0.9 and batch size 256. We set our learning rate of 10-1 so that it linearly warms up from zero within first 5 epochs. A cosine training schedule is applied over the training epochs. We use scale and aspect ratio augmentation with color perturbation as standard data augmentations. We evaluate the performance of trained models on the ImageNet 2012 validation set. The images are resized that the shorter side is of 256 pixels. We then centrally crop 224 x 224 windows from the images as input to measure recognition accuracy.
14

---

Published as a conference paper at ICLR 2020
birdi
bird
rigid kern.
bird
bird
deformable kern.
(a) original
rotation: -90
(c) scaling: 1.5x
Figure 6: Effective Receptive Field Comparison between rigid kernels and DKs under different kinds of Object Deformation. At each row and from left to right, we show the original image (0300x800), the image rotated by -90 degrees and the image scaled by 1.5 times. Images are cropped and resized for the typesetting purpose.
15


Reviews:
[{"review_id": "SkxDm5xCYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Updates:\nThanks for the updates and I appreciate the authors' effort in running new experiments, whose results are very interesting and inspiring to me.  \n\nMy score remains unchanged. The main reason is that I am not an expert in this subject.  \n \n\n--------------------------------------------\nTraditional convolution neural networks are not aware of object’s geometric variations (i.e. rotation), while human being are very good at abstracting out such variation. \nIn this paper, the authors propose an approach known as DKs (deformation kernels) to overcome such issues. The high level picture is make the convolutional filter data dependable. For convnets, the filter is independent of the data. To make it data dependent (potentially able to detect the objects geometric information), DKs first initialize a larger filter (say 9 * 9) that is universal to all inputs and then `subsampling` a smaller filter (say 3 *3), the `subsampling` strategy is input dependent and also learnable (similar to attention mechanism.) \n\nI do not have much background in this field, but I found the ideas of DKs very interesting and novel (assuming this is the first work to make the filter data dependent and learnable.)  I lean to a weakly accept. \n\nMinor Comments: \n1. Below equation (1). Z  -> Z^2; above (1) : j \\in R^2  -> Z^2   \n2. above (5) as a composition --> as a sum ?? \n3. can you elaborate on 'In practice, we use the bilinear sampling operator to interpolate within the discrete kernel grid.' In particular, how the `smalle`r kernel is learned/subsampled from the larger kernel? (i.e. how \\Delta k is learned?) \n\nMore interesting experiments? \n1. Compare performance of the two methods below:   \n a. standard architectures using data deformation (translation, rotation, dilation)\n b. DKs without applying data deformation. \n2. Figure 4.  Could you produce plots similar to the setting of figure 1., i.e. rotate / dilate the images. Showing that DKs could effectively capture such deformation.  \n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1guw8eW9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents the idea of deformable kernels (DKs). As opposed to rigid kernels in standard convolutional networks, DKs allow each of their grid locations to be moved around in a larger kernel field. The offset by which a DK grid cell is moved is computed conditioned on the input to the network. To motivate the idea of DKs, the authors give some background on convolution, receptive and effective receptive fields (ERFs). The authors argue that since ERFs are spatially porous and irregularly distributed, one way to model them is to convolve square grids of input with DKs, which are composed of samples drawn from larger kernels. The authors define the concept of global and local DKs. They further contrast DKs with spatial sampling (deformable convolutions) and argue that although conceptually similar, both approaches are complementary to each other and can be used in combination in practice. Numerical experiments show competitive performance of DKs on image classification and object detection tasks. In the end empirical analysis is performed to analyze the characteristics of DKs.\n\nI am unfamiliar with prior work in this direction, but the idea of DKs seems to be conceptually appealing and as the authors point-out, their approach can be seen as an alternative to spatial sampling for modeling deformations. Unfortunately the authors get hand-wavy when in Sec 3.2, they claim that the idea of subsampling kernels \"roughly generalizes\" to non-linear networks. I don't see how they can generalize what they present beyond piece-wise linear networks. I appreciate the effort to give a background on ERFs and describe (local and global) DKs, but in my opinion, technical sections of the paper partly very obscure. For instance it is not entirely clear how the offset predictors are trained, how exactly the sampling is used, details of architecture etc. \n\nEmpirically the method does not seem to offer a significant performance boost. Also, while the authors sell the idea of subsampling kernels, but the finding that kernel sizes beyond 4x4 don't seem to offer any benefit make the idea practically questionable. \n\nThe idea of DKs seems relevant, but both conceptually and empirically it seems very close to deformable convolutions. The authors need to clearly present their work, including its shortcomings (i.e., generalization or not beyond linear networks).", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eXSFlSqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduce a simple algorithm called deformable kernels. It learns to generate a collection of coordinate offset Δk for each of the convolutional kernel element. Then during convolution, the kernel is treated as a 2D regular grid and sampled (interpolated) according to the generated coordinate offset before applying to the inputs. An auxiliary shallow network is learned to generate those coordinate offsets based in inputs. This method is very similar to the existing \"deformable convolution\" algorithm, though this operate on the kernels instead. Numerical experiments on image classification and object detection tasks show that the method performs better or comparably to strong baselines. It boost the performance even more when combined with existing methods.\n\nThis paper is relatively easy to follow and the ideas are simple and effective.\n\nMy main concern about this paper is the novelty given its similarity to the previous methods. Maybe it could improve if more and in-depth studies are shown that analyze what deformable kernel learns and why that is different from what deformable convolution learn. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In my opinion, this paper is borderline (but my expertise is not in this area) and the reviewers are too uncertain to be of help in making an informed decision.
Decision: Accept (Poster)