Partially dilated convolution

CallMeMisterOwl: <p>Hi there.<br>
I currently have following setup:<br>
<code>nn.Conv1d(IN_SIZE, IN_SIZE, WING * 2 + 1, padding=WING, groups=IN_SIZE)</code><br>
Now I want to implement something I coined partially dilated convolution, in essence I want a convolutional layer that has a “dense island” in the middle. Something like this, where 0: represents a gap due to dilation and 1: a kernel <code>101010101111101010101</code>. Inference is done on the centre element<br>
I was wondering if you could suggest the simplest way to implement this kind of layer?</p>
<p>Thank you in advance for your help!</p>

KFrank: <p>Hi Mister Owl!</p>
<aside class="quote no-group" data-username="CallMeMisterOwl" data-post="1" data-topic="179602" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/callmemisterowl/48/52563_2.png" class="avatar"> CallMeMisterOwl:</div>
<blockquote>
<p>Something like this, where 0: represents a gap due to dilation and 1: a kernel <code>101010101111101010101</code>.</p>
</blockquote>
</aside>
<p>If I understand your use case correctly, the simplest approach would probably<br>
be to instantiate two <code>Conv1d</code>s – one <code>conv_dense</code> with <code>dilation = 1</code>, and a<br>
second <code>conv_dilated</code> with, say, <code>dilation = 2</code>.  Pass the input to your “partially<br>
dilated convolution” layer through both convolutions and then sum the results.</p>
<p>Some of the weights of <code>conv_dilated</code> will overlap with the “dense island”<br>
implemented in <code>conv_dense</code> so there will be some redundancy between<br>
the two sets of weight parameters.  But that should be okay – most neural<br>
networks have some (perhaps less obvious) redundancy and train and<br>
predict just fine.</p>
<p>(You could also store the left dilated wing, the dense island, and the right<br>
dilated wing in three separate trainable parameters, assemble your<br>
“partially dilated” kernel from them, and then call the functional form,<br>
<code>torch.nn.functional.conv1d()</code>, with your assembled kernel.  This<br>
would avoid the redundancy and <em>might</em> prove to run faster, but would<br>
be a bit of a nuisance to implement.)</p>
<p>Best.</p>
<p>K. Frank</p>

CallMeMisterOwl: <p>Hi K. Frank,<br>
thank you for your answer <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"><br>
I will probably just go for the first approach, since I’m short one time, but I will give the second one a try as well, simply because I think it would be a good exercise.</p>
<p>Just to make sure, I understand that approach correctly. One way to achieve this would be to apply a mask to the wings gradients, which zeros every 2nd gradient for the wings (similar to <a href="https://discuss.pytorch.org/t/how-do-i-freeze-the-specific-weights-in-a-layer/104722">here</a>). Essentially doing manual dilation, right ?</p>

KFrank: <p>Hi Mister Owl!</p>
<aside class="quote no-group" data-username="CallMeMisterOwl" data-post="3" data-topic="179602" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/callmemisterowl/48/52563_2.png" class="avatar"> CallMeMisterOwl:</div>
<blockquote>
<p>One way to achieve this would be to apply a mask to the wings gradients, which zeros every 2nd gradient for the wings</p>
</blockquote>
</aside>
<p>I wasn’t imagining doing anything with the gradients (like masking them after<br>
they’ve been computed).  Maybe you could make such a scheme work, but it<br>
would seem quite roundabout to me.</p>
<p>I was suggesting encoding, say, the “left wing” in a trainable parameter,<br>
<code>torch.tensor ([a, b, c, d], requires_grad = True)</code>, and then<br>
“dilating” it into the relevant portion of the kernel you then pass to <code>conv1d()</code>,<br>
<code>torch.tensor ([a, 0, b, 0, c, 0, d, 0, ...])</code>.  Autograd will track<br>
gradients back through your assembled kernel to your trainable parameter<br>
and its <code>a</code>, <code>b</code>, <code>c</code>, <code>d</code> elements.</p>
<p>Best.</p>
<p>K. Frank</p>

CallMeMisterOwl: <p>Hey K. Frank,<br>
that is indeed a clever and clean solution.<br>
I am very grateful for your time and expertise. Thanks again <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>
<p>Best regards</p>