Custom pooling/conv layer

username: deJQK
cooked: <p>If I want to implement some custom pooling or convolution layer, like the gated or tree pooling functions as shown in this <a href="https://arxiv.org/abs/1509.08985" rel="nofollow noopener">article</a>, do I have to modify the c++ <a href="https://github.com/pytorch/pytorch/tree/master/aten/src/THNN/generic" rel="nofollow noopener">code</a>, instead of some simpler extension in python? I found the official website on ‘Custom C++ and CUDA Extensions’, but it seems not sufficient for my goal.</p>

username: mitchus
cooked: <p>I am also interested in this, have you figured it out?</p>

username: deJQK
cooked: <p>Custom convolution layer can be implemented either by inheritance from the class nn.Conv2d, or through the funciton unfold. Unfortunately, unfold do not support dilation.</p>
<p>You may want to check these two topics:</p><aside class="quote" data-post="1" data-topic="5799">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/df788c/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-impelement-a-customized-convolutional-layer-conv2d/5799">How to impelement a customized convolutional layer (conv2d)</a> 
  </div>
  <blockquote>
    Hi, 
I want to implement a customized Conv2d in which some multiplications during the convolution operation are dropped by some probability. This would happen during testing, and preferably different multiplications would drop for different kernels in a layer. Is there an easy way to do this? Do I need to implement the Conv2d function using pytorch functions from scratch? 
Thanks!
  </blockquote>
</aside>
<aside class="quote" data-post="1" data-topic="23697">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dejqk/48/4965_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/general-unfold-function-with-dilation/23697">General unfold function with dilation</a> 
  </div>
  <blockquote>
    PyTorch provide the powerful function unfold, through both <a href="https://pytorch.org/docs/stable/nn.html#id23" rel="noopener nofollow ugc">torch.nn.functional.unfold</a> and the builtin function for tensor <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.unfold" rel="noopener nofollow ugc">torch.Tensor.unfold</a>. It seems the latter is easier to use, and it is more general as it is not restricted to 4D tensor. However, the former implementation support dilation, while the latter does not. 
Does anyone has method to augment the builtin function implementation to make it support dilation?
  </blockquote>
</aside>

<p>Custom pooling can be implemented like this:</p><aside class="onebox githubgist" data-onebox-src="https://gist.github.com/rwightman/f2d3849281624be7c0f11c85c87c1598">
  <header class="source">

      <a href="https://gist.github.com/rwightman/f2d3849281624be7c0f11c85c87c1598" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/rwightman/f2d3849281624be7c0f11c85c87c1598" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/rwightman/f2d3849281624be7c0f11c85c87c1598</a></h4>

  <h5>median_pool.py</h5>
  <pre><code class="Python">import math
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.nn.modules.utils import _pair, _quadruple


class MedianPool2d(nn.Module):
    """ Median pool (usable as median filter when stride=1) module.
    </code></pre>
    This file has been truncated. <a href="https://gist.github.com/rwightman/f2d3849281624be7c0f11c85c87c1598" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: tom
cooked: <aside class="quote no-group" data-username="deJQK" data-post="3" data-topic="18916">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dejqk/48/4965_2.png" class="avatar"> deJQK:</div>
<blockquote>
<p>Unfortunately, unfold do not support dilation.</p>
</blockquote>
</aside>
<p>What is the issue you are seeing?</p>
<pre><code class="lang-auto">inp = torch.randn(1, 3, 10, 12)
w = torch.randn(2, 3, 4, 5)
inp_unf = torch.nn.functional.unfold(inp, (4, 5), dilation=(2,2))
out_unf = inp_unf.transpose(1, 2).matmul(w.view(w.size(0), -1).t()).transpose(1, 2)
out = torch.nn.functional.fold(out_unf, (4, 4), (1, 1))
(torch.nn.functional.conv2d(inp, w, dilation=(2,2)) - out).abs().max().item()
</code></pre>
<p>seems to work for me.</p>
<p>Best regards</p>
<p>Thomas</p>

username: deJQK
cooked: <blockquote>
<p>nn.functional.unfold</p>
</blockquote>
<p>only supports four dim case.</p>

username: tom
cooked: <p>Indeed, from your message, I took the Conv2d class as an indication that you want that. But in the other thread you specifically say non-4d.</p>
<p>Best regards</p>
<p>Thomas</p>

username: deJQK
cooked: <p>Sorry for the ambiguity and thanks for clarification.</p>

username: Aviral_Aggarwal
cooked: <p>I am also trying to implement gated pooling and I am quite badly stuck, have you implemented this?<br>
Help would be greatly appreciated.</p>