My Custom Loss doesn't do backward properly

username: Alessandro_Paticchio
cooked: <p>Hi, I am trying to apply boosting with NN, and to do so I need to multiply the loss of each sample by a scalar weight.<br>
I’m implementing it by slightly changing the BCELoss in the following way:</p>
<p>from torch.nn.modules.loss import BCELoss<br>
import torch<br>
from torch.nn import functional as F</p>
<pre><code class="lang-auto">from torch.nn.modules.loss import BCELoss
import torch
from torch.nn import functional as F


class WeightedBCELoss(BCELoss):
    def __init__(self, weight=None, size_average=None, reduce=None, reduction='mean', weights_boosting=None):
        super(BCELoss, self).__init__(weight, size_average, reduce, reduction)
        self.weights_boosting = weights_boosting

    def forward(self, input, target):
        total_cross_entropy = 0

        for i, sample in enumerate(input):
            y_hat = input[i]
            y = target[i]
            c_e = F.binary_cross_entropy(y_hat, y, weight=self.weight,
                                         reduction=self.reduction).item()
            c_e = c_e * self.weights_boosting[i].item()
            total_cross_entropy = total_cross_entropy + c_e

        return torch.tensor([total_cross_entropy], requires_grad=True)


</code></pre>
<p>But afterwards my model weights do not change at all. Do you have any idea on how to fix it?</p>
<p>N.B. If I try to remove that “requires_grad=True” at the last line, it goes:</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>Thanks in advance</p>

username: albanD
cooked: <p>Hi,</p>
<p>See this other post that is related: <a href="https://discuss.pytorch.org/t/getting-none-on-custom-loss-function/61900" class="inline-onebox">Getting None on custom loss function</a><br>
Does that explains your issue?</p>

username: Alessandro_Paticchio
cooked: <p>Nope, unfortunately… but thanks</p>

username: albanD
cooked: <p>You do the similar thing of rewrapping your result into a Tensor after doing computations on non differentiable elements (python numbers in your case, integer tensors in the other post).<br>
You will have to keep everything into Tensors for the autograd to work.</p>

username: Alessandro_Paticchio
cooked: <p>Mmm ok I think I got the problem, so basically you suggest to work only with tensors and never go to scalar?</p>

username: albanD
cooked: <p>Yes. The autograd engine can only track gradients for Tensors, not other objects unfortunately.</p>

username: Alessandro_Paticchio
cooked: <p>My main problem is using the “total_cross_entropy” value as an accumulator… do you have any idea on how to do it with tensors?</p>

username: albanD
cooked: <p>Taking a quick look at your function, just removing the two <code>.item()</code> and just returning <code>return total_cross_entropy</code> should work no?</p>
<p>If you do <code>0 + t</code> with <code>t</code> a Tensor, you will get a Tensor.</p>

username: Alessandro_Paticchio
cooked: <p>Got it, thank you so much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>