How to difine self defined loss?

username: lzh21cen
cooked: <p>I defined a loss function, it can run normally. however, it produce bad results, while nll can produce accurate results in the same net work. So any one can take look at my code is right?</p>
<p>class CrpsLossExt(Function):<br>
def forward(outEjectF, targetEjectF,nClass):</p>
<pre><code>    targetEjectF = targetEjectF.cpu().numpy()
    predEjectF = outEjectF.cpu().numpy()
   
   # following sentences are not torch function, they are from python,numpy or SCI functions
    predictCdf = utils_heart.real_to_cdf(predEjectF,nClass)
    targetHeavySide = utils_heart.heaviside_function(targetEjectF,nClass)
    crplLoss = utils_heart.crps(predictCdf,targetHeavySide)
    tensorCrplLoss = torch.from_numpy(np.array(crplLoss))
   
    tensorCrplLoss = tensorCrplLoss.requires_grad_()       
    return tensorCrplLoss   

def backward(grad_output): 
    return grad_output</code></pre>

username: KFrank
cooked: <p>Hi lzh21cen!</p>
<p>Please take a look at the “Extending <code>torch.autograd</code>” section<br>
of <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="noopener nofollow ugc">Extending PyTorch</a>.</p>
<aside class="quote no-group quote-modified" data-username="lzh21cen" data-post="1" data-topic="62324" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/b5e925/48.png" class="avatar"> lzh21cen:</div>
<blockquote>
<p>I defined a loss function, it can run normally. however, it produce bad results, while nll can produce accurate results in the same net work.</p>
<p>…</p>
<pre data-code-wrap="python"><code class="lang-python">       # following sentences are not torch function, they are from python,numpy or SCI functions
</code></pre>
</blockquote>
</aside>
<p>Once you move out of pytorch and into something like numpy,<br>
autograd can no longer track your gradients, so you (your loss<br>
function) will have to do it yourself.  That’s what <code>backward</code> is for.</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">    def backward(grad_output): 
        return grad_output
</code></pre>
</blockquote>
</aside>
<p>Your <code>backward</code> doesn’t do anything (except return its input as<br>
its output).  So the gradients that get passed to the optimizer don’t<br>
know anything about the structure of your loss function.  That is,<br>
your gradients are incorrect, so the optimizer won’t be moving<br>
your weights to a lower loss.</p>
<p>You either need to rewrite your loss function using pytorch<br>
<code>Tensor</code> functions so that autograd can track and calculate<br>
the gradients automatically for you, or you have do calculus<br>
on your loss function to get its gradients and implement them<br>
by hand in your <code>backward</code> function.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: lzh21cen
cooked: <p>thanks a lot, K. Frank<br>
If I define loss function in another way which will be inherited from nn.Module, must I use torch functions in all forward steps?</p>
<p>class CrpsLossModule(nn.Module):<br>
def <strong>init</strong>(self, nClass, reduce=True):<br>
super(CrpsLossModule, self).<strong>init</strong>()<br>
self.nClass = nClass<br>
self.reduce = reduce</p>
<pre><code>def forward(self, outEjectF, targetEjectF):
    
    # forward code here

    if self.reduce:
        return torch.mean(F_loss)
    else:
        return F_loss</code></pre>

username: KFrank
cooked: <p>Hello lzh21cen!</p>
<aside class="quote no-group" data-username="lzh21cen" data-post="3" data-topic="62324" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/b5e925/48.png" class="avatar"> lzh21cen:</div>
<blockquote>
<p>If I define loss function in another way which will be inherited from nn.Module, must I use torch functions in all forward steps?</p>
</blockquote>
</aside>
<p>You could try it out and see.  Pass a <code>requires_grad = True</code> tensor<br>
through your loss function and see if <code>.backward()</code> gives the correct<br>
result.  Something like:</p>
<pre data-code-wrap="python"><code class="lang-python">input = torch.randn ((2, 5), requires_grad=True)
target = ...
loss = my_custom_loss_function (input, target)
loss.backward()
print (input.grad)
</code></pre>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">        # forward code here

        if self.reduce:
            return torch.mean(F_loss)
        else:
            return F_loss
</code></pre>
</blockquote>
</aside>
<p>(As a side note, I have no idea what <code>F_loss</code> means here, so I have<br>
no idea whether this code makes sense or would work the way you<br>
want.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: lzh21cen
cooked: <p>Thanks, your answers are extremely helpful! I think I had to look in detail the documentation.</p>