How to define a new layer with autograd?

username: big_tree
cooked: <p>Hi all,</p>
<p>I am new to pytorch and also new to autograd.  My question is : Do I need to compute the partial derivatives for my functions parameters?<br>
For example: My new layer want to compute a 1-d gaussian probability density value, the function is,<br>
<code>f(x)=a * exp^((x-b)^2 / c))</code> ,<br>
where <code>a,b,c</code> are the parameters need to be updated. I think all of these operations are basic operation and the output is a scalar. Do I still need to write backward code like we do in Caffe?  Or May I just define a new module only with forward function then pytorch will compute the parameters’ derivatives automatically for me?</p>

username: apaszke
cooked: <p>Sure, this will be handled for you. For example:</p>
<pre><code class="lang-python">import torch.nn as nn
from torch.autograd import Variable

class Gaussian(nn.Module):
    def __init__(self):
        self.a = nn.Parameter(torch.zeros(1))
        self.b = nn.Parameter(torch.zeros(1))
        self.c = nn.Parameter(torch.zeros(1))

    def forward(self, x):
        # unfortunately we don't have automatic broadcasting yet
        a = self.a.expand_as(x)
        b = self.b.expand_as(x)
        c = self.c.expand_as(x)
        return a * torch.exp((x - b)^2 / c)

module = Gaussian()
x = Variable(torch.randn(20))
out = module(x)
loss = loss_fn(out)
loss.backward()

# Now module.a.grad should be non-zero.
</code></pre>

username: big_tree
cooked: <p>Thank you so much for your kind help! I still would like to ask few questions:<br>
(1) should we add   <code>super(Gaussian, self).__init__()</code> after <code>def __init__(self):</code>?<br>
(2) After we add this new module to our network and start to train it, Do I only need to use</p>
<blockquote>
<p><code>optimizer = optim.SGD(net.parameters(), lr = 0.01) optimizer.zero_grad()  output = net(input) loss = criterion(output, target) loss.backward() optimizer.step() </code></p>
</blockquote>
<p>to update all the parameters?</p>

username: apaszke
cooked: <ol>
<li>My bad, I have typed the example quickly just to give you a sense of how the code should look like, so I forgot about the super call. Nice catch!</li>
<li>Yes, that should do it. Since <code>a</code>, <code>b</code> and <code>c</code> are <code>nn.Parameter</code> objects, that were assigned directly to a module, they should appear in <code>.parameters()</code> of the main container.</li>
</ol>

username: big_tree
cooked: <p>Thank you so much for your help!</p>

username: jdhao
cooked: <p>Now we have defined the Gaussian layer, I want to know how to use it in a network of several layers. For example,</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.fc1 = nn.Linear(100, 20)
        self.gauss = Gaussian()
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.gauss(x)

net = MyNet()
</code></pre>
<p>Is the above code correct? Or do we have to set up other things in order to use the newly-defined layer in other network?</p>

username: smth
cooked: <p>the above is correct.</p>

username: Soumava_Roy
cooked: <p>Can I define my own layer with a forward/backward function or I will have to define the layer as a Function so that I can define the backward function too?</p>

username: Hao_Chu
cooked: <p>You have no need to define the backward function yourself as long as the functions you are using to calculate the loss function is in PyTorch scope. You should definitely define a forward function just as shown by <a class="mention" href="/u/jdhao">@jdhao</a>.</p>

username: herleeyandi
cooked: <p>By the way if we implement a layer and we have scipy or numpy operation inside of it <a href="http://pytorch.org/tutorials/advanced/numpy_extensions_tutorial.html" rel="nofollow noopener">Like in here</a>, can it be run and accelerate on GPU?, or the layer just run on CPU because our numpy and scipy can not run on GPU?</p>

username: jdhao
cooked: <p>you can add <code>.cuda()</code> to both the module and the <code>input</code> to see if the given example can run without any error.</p>

username: herleeyandi
cooked: <p><a class="mention" href="/u/jdhao">@jdhao</a> Yes it works I have tried it before. However I just want all operation works on GPU. In that example I think (please correct me if I am wrong) it still can run but the operation which run on GPU is just tensor operation like in <a href="http://pytorch.org/docs/master/tensors.html" rel="nofollow noopener">here</a> or some basic pytorch function. So when we use <code>.cuda()</code> it will have big communication cost between GPU and CPU. For example when I am trying manual convolution using python loop operation to pytorch tensor it needs so much time, compare with ready made pytorch convolution. I just want to know what I miss in extending pytorch, whether all operation even when using <code>scipy</code> declared in pytorch can really work on GPU or scipy process still works on CPU but the tensor on GPU which give big communication cost between CPU and GPU.</p>

username: jdhao
cooked: <p>Most of the time we do not need to extend PyTorch using numpy. As long as you use the builtin method of Variable, you can only write forward method and backward gradient computation is handled by autograd. So using a composition of builtin Variable method to achieve what you want is more time-saving.</p>

username: colesbury
cooked: <p><a class="mention" href="/u/herleeyandi">@herleeyandi</a>, the portions of your code which use <code>scipy</code> will not be GPU-accelerated. Only operations on CUDA Torch tensors and Variables will be GPU-accelerated.</p>

username: herleeyandi
cooked: <p><a class="mention" href="/u/colesbury">@colesbury</a> I see, so how about if I want to create some functions which is GPU accelerated?, should I use CFFI which coded using CUDA C++ ?, Can you give me more hint <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: colesbury
cooked: <p>Yes, you can do that. Writing new CUDA kernels usually requires a lot of effort. If you can express your layer in terms of existing Tensor operations, then that’s usually a better way to get started. If you can’t do that, then you might have to write new kernels.</p>

username: MahdiNazemi
cooked: <p>What if the forward function is not differentiable? How should I update the parameters and return the gradient w.r.t inputs?</p>

username: jwang9
cooked: <p>I tried the piece of code posted by apaszke, but I got the following error.</p>
<pre><code class="lang-auto">RuntimeError: bitwise_xor(): functions with out=... arguments don't support automatic differentiation, but one of the arguments requires grad.
</code></pre>
<p>So I am confused how <a class="mention" href="/u/big_tree">@big_tree</a> and <a class="mention" href="/u/jdhao">@jdhao</a> managed to get this code to work.</p>
<p>I read the documentation <a href="https://pytorch.org/docs/master/notes/extending.html" rel="noopener nofollow ugc">Extending PyTorch</a>, but the extending torch.nn (adding a module) section using the LinearFunction defined above, which explicitly defines both forward and backward functions, but this doesn’t seem to be the case in apaszke 's sample code above. So there is a way to make his code work, without explicitly encoding the gradient myself?</p>

username: jwang9
cooked: <p>I figured that the problem arises from the power calculation in python. I changed ^2 to **2 and it works now.</p>