Using mathematical expression over convolution kernels

nclk: <p>Hi,</p>
<p>I want to use some mathematical expression over kernels in convolution. I wrote some codes as a dummy example.</p>
<p>1- My question is this kind of definition is true for gradient flow?, and<br>
2- Should we use <code>super()__init__</code>  definition?<br>
3- Does autograd allow logical constrains like x = x.*(x&lt;5) (matlab-like) in forward?</p>
<pre><code class="lang-auto">class new_conv(nn.Conv2d):
    def __init__(self, dim):
        self.W   = Variable(torch.randn(dim, 1).type(dtype), requires_grad=True)
        self.act = F.relu()

    def forward(self, x):
        xMean = x.mean(dim=1, keepdim=True).mean(dim=2, keepdim=True).mean(dim=3, keepdim=True)
        x = x - xMean 
        
        w = self.W**2
        wMean = w.mean(dim=1, keepdim=True).mean(dim=2, keepdim=True).mean(dim=3, keepdim=True)
        w = F.relu(w-wMean)
        x = torch.cat((x,x**2),2)              # Channelwise concatenating 
        
        return self.act(F.conv2d(x, w))
</code></pre>
<p>4- Last question, can we express this module as function? In such a definition, every iteration I use new_conv in the forward step, I doubt that weights are reassigned. So even if the expression works, it will be untrainable, as if.</p>
<p>Many thanks…</p>
<pre><code class="lang-auto">def new_conv(x,dim):
    W   = Variable(torch.randn(dim, 1).type(dtype), requires_grad=True)
    xMean = x.mean(dim=1, keepdim=True).mean(dim=2, keepdim=True).mean(dim=3, keepdim=True)
    x = x - xMean 
        
    w = W**2
    wMean = w.mean(dim=1, keepdim=True).mean(dim=2, keepdim=True).mean(dim=3, keepdim=True)
    w = F.relu(w-wMean)
    x = torch.cat((x,x**2),2)              # Channelwise concatenating 
        
    return F.relu(F.conv2d(x, w))
</code></pre>

albanD: <p>Hi,</p>
<ol>
<li>Not sure what you refer to in that question.</li>
<li>You should always call <code>super().__init__()</code> if you subclass an <code>nn.Module()</code> and you write a custom <code>__init__()</code> function.</li>
<li>The autograd allows for that. Just note that no gradient (a gradient of 0) will flow back on the side of <code>x&lt;5</code> because this gives a piece-wise constant output which has a gradient of 0 almost everywhere. In general, the autograd engine will raise an exception if it cannot compute the gradients you ask for.</li>
<li>You want to use <code>nn.Module</code> over just a python function if you have some parameters. Which is your case here actually. Note as well that parameters should be of type <code>nn.Parameter</code> (in your case: <code>self.W = nn.Parameter(torch.randn(dim, 1).type(dtype))</code> so that it will be properly found as a parameter of your network when you do <code>net.parameters()</code> (for example to give all your parameters to your optimizer).<br>
If you do the function you showed, then it will generate random weights every time and no learning will happen, which is not what you want I guess as you set requires_grad = True here.<br>
Keep in mind that in pytorch, your code is run at <strong>every</strong> forward pass.</li>
</ol>

nclk: <p><a class="mention" href="/u/alband">@albanD</a> Thank you for your answer. it was informative and helpful for me.</p>