Partial derivative of custom loss

username: hyuntae
cooked: <p>Hi, I have a problem to implement custom loss using torch.autograd.Function</p>
<p>I know that there is MSELoss() in pytorch.<br>
But I want to implement it via custom loss with image data.</p>
<pre><code class="lang-auto">class customMSE(torch.autograd.Function):
    def forward(ctx,output, target):
        ctx.save_for_backward(output, target)

        loss = torch.norm(output-target,p=2) / 2
    
    def backward(ctx, grad_output):
        output, target = ctx.saved_tensors

        return (output - target) * partial_derivative
 
</code></pre>
<p>In this case, how can I define partial derivative of output with respect to the network parameters.<br>
Should I have to implement all the chain rule?<br>
Is there any method to get the partial derivative?</p>

username: anantguptadbl
cooked: <p>Hello <a class="mention" href="/u/hyuntae">@hyuntae</a></p>
<p>I had a followup question</p>
<ul>
<li>Do you want to just have a custom Loss Function between input and output</li>
</ul>
<pre><code class="lang-auto">def lossCalc(x,y):
    return torch.sum(torch.add(x,-y)).pow(2)
</code></pre>
<pre><code class="lang-auto">.....
# Input Data
X=np.random.rand(numRows * numSentences * numWords,inputDim)

# Resizing
X=X.reshape(totalBatches,numSentences,numWords,inputDim)

# Creating Y
Y=np.random.randint(2,size=(numRows,1))

for epoch in range(epochRange):
    lossVal=0
    for curBatch in range(totalBatches):
        model.zero_grad()
        dataInput=torch.autograd.Variable(torch.Tensor(X[curBatch]))
        dataOutput=model(dataInput)
        loss=lossCalc(dataOutput,torch.Tensor(Y[curBatch]))
        loss.backward()
        lossVal = lossVal + loss
        optimizer.step()
    if(epoch % 1==0):
        print("For epoch {}, the loss is {}".format(epoch,lossVal))
print("Model Training completed")
</code></pre>
<ul>
<li>Or do you want to use some operation on the gradients<br>
If this is the case, how will you know the expected value for comparison?</li>
</ul>

username: Thanhhuan_Nguyen
cooked: <p>Vào 16:14, T.2, 25 Th11, 2019 Hyuntae Choi via PyTorch Forums <a href="mailto:noreply@discuss.pytorch.org">noreply@discuss.pytorch.org</a> đã viết:</p>

username: hyuntae
cooked: <p>Thanks for reply @ <a href="https://discuss.pytorch.org/u/anantguptadbl">anantguptadbl</a>.</p>
<p>My example code is just compare between input and output.</p>
<p>I think that torch.autograd calculates the partial derivative of output.<br>
In this case <img src="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef68483839e11943830105fc9f96d5462247f99c.gif" alt="CodeCogsEqn" data-base62-sha1="y9TxpeEeqhDbVUFfCszOqAuucqM" width="197" height="39"><br>
where u is  output of network, f is input image and w is parameter of network.</p>
<p>But I don’t know how to calculate  <img src="https://discuss.pytorch.org/uploads/default/original/3X/9/4/9492a75376c948016adfb6ab96379f4118d5e0cd.gif" alt="1" data-base62-sha1="lckRUyHdfWWqJceuEYXBJ465dcV" width="24" height="39">  in my custom loss backward function.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Your custom loss only needs to compute the partial derivative wrt it’s input. The autograd engine will use the chain rule and the other gradients formulas implemented in pytorch to compute the gradienst wrt the weights.</p>
<p>What you are given, if your loss is L and you compute out = your_fn(inp), is grad_output is dL/dout. And you should return dL/dinp. That is usually written as dL/dinp = dL/dout * dout/dinp. which is the formula you present above.<br>
The autograd will take care of computing dL/dw using your computed dL/dinp.</p>

username: mathematics
cooked: <p>Hi , how did you solved problem? would you please explain whole custom loss how can be calculate with nice example.<br>
Alban said  we have to compute derivative of  loss wrt input (dl/dinp) when using autograd but in tutorials custom relu function  why we didnt calculated <code>grad_input[input &gt; 0 ] = 1</code>.please would you give example on mse loss as you solved . Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>