Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper introduce a simple algorithm called deformable kernels. It learns to generate a collection of coordinate offset Δk for each of the convolutional kernel element. Then during convolution, the kernel is treated as a 2D regular grid and sampled (interpolated) according to the generated coordinate offset before applying to the inputs. An auxiliary shallow network is learned to generate those coordinate offsets based in inputs. This method is very similar to the existing "deformable convolution" algorithm, though this operate on the kernels instead. Numerical experiments on image classification and object detection tasks show that the method performs better or comparably to strong baselines. It boost the performance even more when combined with existing methods.

This paper is relatively easy to follow and the ideas are simple and effective.

My main concern about this paper is the novelty given its similarity to the previous methods. Maybe it could improve if more and in-depth studies are shown that analyze what deformable kernel learns and why that is different from what deformable convolution learn. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This work presents the idea of deformable kernels (DKs). As opposed to rigid kernels in standard convolutional networks, DKs allow each of their grid locations to be moved around in a larger kernel field. The offset by which a DK grid cell is moved is computed conditioned on the input to the network. To motivate the idea of DKs, the authors give some background on convolution, receptive and effective receptive fields (ERFs). The authors argue that since ERFs are spatially porous and irregularly distributed, one way to model them is to convolve square grids of input with DKs, which are composed of samples drawn from larger kernels. The authors define the concept of global and local DKs. They further contrast DKs with spatial sampling (deformable convolutions) and argue that although conceptually similar, both approaches are complementary to each other and can be used in combination in practice. Numerical experiments show competitive performance of DKs on image classification and object detection tasks. In the end empirical analysis is performed to analyze the characteristics of DKs.

I am unfamiliar with prior work in this direction, but the idea of DKs seems to be conceptually appealing and as the authors point-out, their approach can be seen as an alternative to spatial sampling for modeling deformations. Unfortunately the authors get hand-wavy when in Sec 3.2, they claim that the idea of subsampling kernels "roughly generalizes" to non-linear networks. I don't see how they can generalize what they present beyond piece-wise linear networks. I appreciate the effort to give a background on ERFs and describe (local and global) DKs, but in my opinion, technical sections of the paper partly very obscure. For instance it is not entirely clear how the offset predictors are trained, how exactly the sampling is used, details of architecture etc. 

Empirically the method does not seem to offer a significant performance boost. Also, while the authors sell the idea of subsampling kernels, but the finding that kernel sizes beyond 4x4 don't seem to offer any benefit make the idea practically questionable. 

The idea of DKs seems relevant, but both conceptually and empirically it seems very close to deformable convolutions. The authors need to clearly present their work, including its shortcomings (i.e., generalization or not beyond linear networks).
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Updates:
Thanks for the updates and I appreciate the authors' effort in running new experiments, whose results are very interesting and inspiring to me.  

My score remains unchanged. The main reason is that I am not an expert in this subject.  
 

--------------------------------------------
Traditional convolution neural networks are not aware of object’s geometric variations (i.e. rotation), while human being are very good at abstracting out such variation. 
In this paper, the authors propose an approach known as DKs (deformation kernels) to overcome such issues. The high level picture is make the convolutional filter data dependable. For convnets, the filter is independent of the data. To make it data dependent (potentially able to detect the objects geometric information), DKs first initialize a larger filter (say 9 * 9) that is universal to all inputs and then `subsampling` a smaller filter (say 3 *3), the `subsampling` strategy is input dependent and also learnable (similar to attention mechanism.) 

I do not have much background in this field, but I found the ideas of DKs very interesting and novel (assuming this is the first work to make the filter data dependent and learnable.)  I lean to a weakly accept. 

Minor Comments: 
1. Below equation (1). Z  -> Z^2; above (1) : j \in R^2  -> Z^2   
2. above (5) as a composition --> as a sum ?? 
3. can you elaborate on 'In practice, we use the bilinear sampling operator to interpolate within the discrete kernel grid.' In particular, how the `smalle`r kernel is learned/subsampled from the larger kernel? (i.e. how \Delta k is learned?) 

More interesting experiments? 
1. Compare performance of the two methods below:   
 a. standard architectures using data deformation (translation, rotation, dilation)
 b. DKs without applying data deformation. 
2. Figure 4.  Could you produce plots similar to the setting of figure 1., i.e. rotate / dilate the images. Showing that DKs could effectively capture such deformation.  


review_title: Official Blind Review #3

Meta-review: In my opinion, this paper is borderline (but my expertise is not in this area) and the reviewers are too uncertain to be of help in making an informed decision.
Decision: Accept (Poster)