Deformable convolution layer with 'variable filter'

username: shirui-japina
cooked: <p>I think that <a href="https://arxiv.org/abs/1703.06211" rel="noopener nofollow ugc">deformable convolution</a> is a relatively creative idea.<br>
But what I feel is insufficient is that the size of the convolution kernel cannot be changed.<br>
How to implement such a convolution layer in PyTorch:</p>
<ol>
<li>
<p><strong>Deformable convolution.</strong></p>
</li>
<li>
<p><strong>Variable convolution kernel size.</strong></p>
</li>
<li>
<p><em>(If point 2 does not make sense in mathematical theory,)</em><br>
<strong>Variable size for the image which put into the convolution kernel.</strong><br>
<em>(When putting into the convolution kernel, the image will be resized to adapt to the convolution layer.)</em></p>
</li>
</ol>
<p>I mean, how to implement such a convolution layer in PyTorch:</p>
<ul>
<li>point 1 and point 2</li>
</ul>
<p>or</p>
<ul>
<li>point 1 and point 3</li>
</ul>
<p>For the point 3, add a note.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c2ea7830a36d833509108549750dbe237c03020.jpeg" alt="v2-c9b00043ba326451979abda5417bfcdf_hd" data-base62-sha1="d9tSN8y4Bec9dhwLs2V0sISZHeo" width="437" height="345"><br>
My question is whether the size of <em><strong>the convolution kernel</strong></em> can also <strong>be changed dynamically</strong>.<br>
If it doesn’t make sense in mathematical theory, I think it’s useful to dynamically <em><strong>adjust the size of the cut image</strong></em>. Then I mean <em><strong>the cut image</strong></em> is <em><strong>before</strong></em> the image <strong>with source pixel</strong> as the picture shows. <em><strong>The cut image</strong></em> will be <strong>changed the size</strong> (the same as the size of the image <strong>with source pixel</strong> in the picture) and put into the convolution kernel.</p>

username: albanD
cooked: <p>Hi,</p>
<ol>
<li>
<p>Could you explain exactly what a “Deformable Convolution” should compute? And what are the parameters?</p>
</li>
<li>
<p>You can do variable kernel size by doing a custom module that computes the new kernel during the forward pass and then calls the convolution operation with the new kernel size.</p>
</li>
<li>
<p>Here again you can do a custom module that will change the input size by interpolating the new values.</p>
</li>
</ol>

username: shirui-japina
cooked: <p>Thanks for your reply.</p>
<aside class="quote no-group" data-username="albanD" data-post="2" data-topic="58799" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<ol>
<li>Could you explain exactly what a “Deformable Convolution” should compute? And what are the parameters?</li>
</ol>
</blockquote>
</aside>
<ul>
<li>What a “Deformable Convolution” should compute.</li>
</ul>
<p>I want to deal with CT images of lung nodules and <strong>predict its benign and malignant</strong> like the image below. The point is that features in physiological medical images tend to <strong>vary widely</strong>. So I want to use “Deformable Convolution” to adapt to that.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/8/780c3760e9e6c4a056b6ce0d7a854aaf5a52ee62.png" data-download-href="https://discuss.pytorch.org/uploads/default/780c3760e9e6c4a056b6ce0d7a854aaf5a52ee62" title="whole_68"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/780c3760e9e6c4a056b6ce0d7a854aaf5a52ee62_2_250x250.png" alt="whole_68" data-base62-sha1="h7Zt7VuPHRewvPlWAilodwKB71g" width="250" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/780c3760e9e6c4a056b6ce0d7a854aaf5a52ee62_2_250x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/780c3760e9e6c4a056b6ce0d7a854aaf5a52ee62_2_375x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/780c3760e9e6c4a056b6ce0d7a854aaf5a52ee62_2_500x500.png 2x" data-dominant-color="6C6C6C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">whole_68</span><span class="informations">512×512 172 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It is difficult to determine <strong>the size of the image cut</strong>.<br>
As shown below (size of 16 * 16, 32 * 32, 48 * 48), too big or too small is not appropriate.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/2/621b2aacd6169cdfca7406703c4d8c7f3d903f5e.png" alt="patch_426_286_68_16" data-base62-sha1="dZSZMvqBsdKCKcma44qgZDF2fSK" width="16" height="16"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/6/0/60298c783fe2ba5844331c4f7ee1132f77fd99bb.png" alt="patch_426_286_68_32" data-base62-sha1="dIGR05Tui8Plz4ZMRP2Zv5dEIAz" width="32" height="32"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/d/f/df5e870157063b4289f92c5438ffe4097025faa2.png" alt="patch_426_286_68_48" data-base62-sha1="vS0ZZrb98MgBojE2biQBK1lk8kq" width="48" height="48"></p>
<ul>
<li>What are the parameters.
<ul>
<li>the size of image cut. (I think the variable size is the best.)<br>
In fact, it is difficult to figure out the best segmentation size artificially.</li>
<li>the size of kernel. (I think the variable size is the best.)<br>
(The point is that features in physiological medical images tend to <strong>vary widely</strong>.)</li>
<li>weights in the kernel. (There must be.)</li>
<li>offset of the convolution kernel. (In fact, it’s the offset parameters of the “Deformable Convolution”)</li>
</ul>
</li>
</ul>
<aside class="quote no-group" data-username="albanD" data-post="2" data-topic="58799" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<ol start="2">
<li>You can do variable kernel size by doing a custom module that computes the new kernel during the forward pass and then calls the convolution operation with the new kernel size.</li>
</ol>
</blockquote>
</aside>
<p>I’m sorry, I do not understand. If the kernel size is variable, I mean, they will be independent.<br>
Suppose there are three different sizes of convolution kernels: A = (3 * 3), B = (5 * 5), C = (7 * 7).<br>
When the model thinks A is suitable, then it will use kernel A to train and update the weights in A. But it won’t update the weights in B and C. This is my understanding.</p>
<ul>
<li>Can such model work well?</li>
<li>How can we code a model like this in PyTorch? (<code>if...else...</code> in <code>forward</code> function?)</li>
<li>Or is my understanding wrong?</li>
</ul>
<aside class="quote no-group" data-username="albanD" data-post="2" data-topic="58799" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<ol start="3">
<li>Here again you can do a custom module that will change the input size by interpolating the new values.</li>
</ol>
</blockquote>
</aside>
<p>I’m so sorry, I don’t understand…</p>

username: albanD
cooked: <p>You can have a model with conditionals with no problem in pytorch:</p>
<pre><code class="lang-auto">class MyMod(nn.Module):
  def __init__(self, params):
    self.A = MyModA()
    self.B = MyModB()
    self.C = MyModC()

  def forward(self, input):
    cond = compute_condition(input)
    if cond == 0:
      return self.A(input)
    elif cond == 1:
      return self.B(input)
    else:
      return self.C(input)
</code></pre>
<p>Now whether such model will work well for your task I don’t know. I’m afraid you will have to try  and see experimentally if it does.</p>

username: shirui-japina
cooked: <p>Thanks for your suggestion.<img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"><img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"> I will try it.<img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: ahkarami
cooked: <p><a class="mention" href="/u/alband">@albanD</a> &amp; <a class="mention" href="/u/shirui-japina">@shirui-japina</a><br>
There is a great reference for understanding Deformable Convolution Operator:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://d3g6anj9jkury9.cloudfront.net/assets/favicons/favicon-16x16-c1207cd2f3a20fd50de0e585b4b307a3.png" class="site-icon" width="16" height="16">
      <a href="https://leanpub.com/convolutions-for-deep-learning" target="_blank" rel="nofollow noopener">Leanpub</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:95/124;"><img src="https://d2sofvawe08yqg.cloudfront.net/convolutions-for-deep-learning/medium?1575064090" class="thumbnail"></div>

<h3><a href="https://leanpub.com/convolutions-for-deep-learning" target="_blank" rel="nofollow noopener">Deep Dive into Different Types of Convolutions for Deep Learning</a></h3>

<p>This book teaches the different types of convolution operators to design Deep Neural Networks with a variety of illustrative figures and examples. The basic concepts of the convolution operator to its advanced types are described with full details in...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>