Depthwise deformable convolutions are slow

username: MauroPfister
cooked: <p>Hi</p>
<p>I’m working on a project where I use deformable convolutions to shift features in the spatial dimension. Since I only want to shift them spatially it seemed logical to use depthwise convolutions which can be done by using <code>groups = in_channels = out_channels</code> as far as I understand.</p>
<p>Unfortunately the deformable convolutions implemented in <code>torchvision.ops.DeformConv2d</code> as well as the ones implemented in <a href="https://github.com/open-mmlab/mmdetection" rel="nofollow noopener">mmdetection</a> are very slow when using <code>groups = in_channels</code> (see time measurements below).</p>
<p>I’d like to avoid using full convolution kernel (<code>3x3xin_channels</code>) because I don’t want to increase the capacity of my network too much. Is there a way to a) either speed up the deformable convolutions with group option, or b) use the same <code>3x3</code> kernel for each of the input channels?</p>
<p>Thanks in advance =)</p>
<p>Script used to measure times: <a href="https://gist.github.com/MauroPfister/6643169d1b0d9b656277dbaaa8548745" rel="nofollow noopener">https://gist.github.com/MauroPfister/6643169d1b0d9b656277dbaaa8548745</a></p>
<pre><code class="lang-auto"># Output of deform_conv_speed.py

bs: 01, in-channels: 512, groups: 1
Time normal conv:                0.35 ms
Time torchvision deform conv:    0.37 ms
Time mmdet deform conv:          0.38 ms
----------------------------------------
bs: 01, in-channels: 512, groups: 1
Time normal conv:                0.35 ms
Time torchvision deform conv:    0.34 ms
Time mmdet deform conv:          0.31 ms
----------------------------------------
bs: 16, in-channels: 512, groups: 512
Time normal conv:                0.31 ms
Time torchvision deform conv:   15.46 ms
Time mmdet deform conv:         14.05 ms
----------------------------------------
bs: 16, in-channels: 512, groups: 512
Time normal conv:                0.35 ms
Time torchvision deform conv:   15.46 ms
Time mmdet deform conv:         14.06 ms
----------------------------------------
</code></pre>

username: ptrblck
cooked: <p>You have a typo in your profiling script and are using the <code>mmdetection</code> implementation twice instead of the <code>torchvision</code> one. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"><br>
That being said, you can see the <code>torchvision</code> CUDA implementation <a href="https://github.com/pytorch/vision/blob/8c9585540a3e71f73ebc281445e888aa841824bd/torchvision/csrc/cuda/DeformConv_cuda.cu#L131-L186">here</a>, which uses an im2col approach and applies the bilinear interpolation.</p>
<p>Note that “normal” convs are heavily accelerated for “typical” shapes in cudnn and an im2col approach would not yield the same performance.<br>
The paper mentions</p>
<blockquote>
<p>Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from the target tasks, without additional supervision.</p>
</blockquote>
<p>which seems to learn the offsets and thus selecting a fast kernel for a specific workload seems to not work out of the box. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>

username: MauroPfister
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for having a look at my script and spotting the typo! I updated both the profiling output and the script.</p>
<p>I already feared that it would not be straight forward to optimize for the grouped convs case. A possible alternative for me would be a applying the same <code>3x3</code> kernel to all channels. I read that something like this can be done by merging batch and channel dimensions together, performing the conv and then splitting them again. However, in my case I have different offsets for each image in a batch, so I cannot perform the exact same convolution over all channels in a batch (even though the kernel would be the same).</p>
<p>Do you have any idea how to solve this?</p>

username: ptrblck
cooked: <p>Yes, the dimension wrapping would work.<br>
Could you explain what the offsets are exactly doing, as I’m not deeply familiar with the implementation?</p>

username: MauroPfister
cooked: <p>The offsets determine the sampling locations of the kernel at each point in the output map. <a href="https://towardsdatascience.com/review-dcn-deformable-convolutional-networks-2nd-runner-up-in-2017-coco-detection-object-14e488efce44" rel="nofollow noopener">This article</a> explains it very well (especially the first image). For example a <code>3x3</code> deformable convolution on a <code>(h, w)</code> input has an “offset map” of <code>(18, h, w)</code>. 18 because <code>9 x (x,y)</code> coordinates for the sampling locations.</p>
<p>These offset maps are calculated with another small CNN based on the input, so they are different for every image in the batch.</p>
<p>I hope this makes things a bit more clear =)</p>

username: MauroPfister
cooked: <p>I think I found a rather ugly hack to increase the performance of my network.</p>
<p>If I understood the concept correctly, a grouped (depthwise) convolution is essentially a normal convolution where only the i-th channel of the i-th kernel contains non zero weights. That means I can train my network using grouped convolutions (<code>groups = in_channels</code>) and during inference I use full convolutions (<code>groups = 1</code>) but I initialize the kernels with 0 and insert the previously trained kernels at the i-th channel of the i-th kernel. I think the code snippet below illustrates better what I’m doing.</p>
<pre><code class="lang-auto"># model.deform_conv.weight.shape = [512, 512, 3, 3]
# pretrained_kernels.shape = [512, 1, 3, 3]

for i in range(model.deform_conv.weight.shape[0]):
    model.deform_conv.weight[i, i, :, :] = pretrained_kernels[i, 0, :, :]
</code></pre>
<p>I modified my code accordingly and can confirm that the results are exactly the same as with grouped convolutions but the inference time is decreased by a factor 8.</p>
<p>Now I’m wondering if this special case of grouped convolutions is really that hard to implement? Couldn’t the “hack” that I explained above also be implemented on CUDA level?</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> Maybe you can tell me if this is feasible or not? Thanks a lot!</p>

username: ptrblck
cooked: <p>That’s a neat trick! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Are you recreating these “zero-padded” convolutions in each iteration or are you zeroing out the gradients?<br>
Did you also verify that you get the same gradients in both approaches?</p>

username: MauroPfister
cooked: <p>For the moment I use this approach only during inference. That is, I train with grouped convolutions (which does not seem to be much slower) and modify the network afterwards for faster inference.</p>
<p>I did not think about using the same approach during training, but it is a great idea! Do you think it would be sufficient to initialize theses “zero-padded” kernels once in the beginning because all the zero weights would generate a zero gradient?<br>
If that does not work, at which point would I have to zero out the gradients? Right before taking the optimizer step?</p>

username: MauroPfister
cooked: <p>Just for the sake of completeness: The profiling times in my first post are not correct, but I cannot edit the post anymore. Here are the correct ones:</p>
<pre><code class="lang-auto"># Output of deform_conv_speed.py

bs: 01, in-channels: 512, groups: 1
Time normal conv:                0.37 ms
Time torchvision deform conv:    0.37 ms
Time mmdet deform conv:          0.36 ms
----------------------------------------
bs: 01, in-channels: 512, groups: 512
Time normal conv:                0.19 ms
Time torchvision deform conv:   15.23 ms
Time mmdet deform conv:         13.67 ms
----------------------------------------
bs: 16, in-channels: 512, groups: 1
Time normal conv:                1.15 ms
Time torchvision deform conv:    2.26 ms
Time mmdet deform conv:          2.19 ms
----------------------------------------
bs: 16, in-channels: 512, groups: 512
Time normal conv:                0.32 ms
Time torchvision deform conv:   15.06 ms
Time mmdet deform conv:         13.76 ms
----------------------------------------
</code></pre>

username: MauroPfister
cooked: <blockquote>
<p>Are you recreating these “zero-padded” convolutions in each iteration or are you zeroing out the gradients?<br>
Did you also verify that you get the same gradients in both approaches?</p>
</blockquote>
<p>I tried implementing <a class="mention" href="/u/ptrblck">@ptrblck</a>’s  idea and checking it on a <a href="https://gist.github.com/MauroPfister/79e9b500a93c028ff16fc12527763c6e" rel="noopener nofollow ugc">toy example</a>. Currently I’m using <code>torch.equal()</code> to compare the gradients. They do not match up in 100% of the cases but almost always. I’m not sure if this is due to some numerical errors or if there is something wrong in my script.</p>