Own BCELoss implementation gradients deviate slightly from pytorch version

username: Erwin_van_Thiel
cooked: <p>Hi, I have a question regarding a custom loss function. I am trying to design a loss function that suits my purpose (its related to adversarial attacks). I want to create a function that is based on BCELoss() so I started off with my own BCELoss() implementation:</p>
<pre><code class="lang-auto">class MlaLoss(nn.Module):
    def __init__(self, weight=None, size_average=True):
        super(MlaLoss, self).__init__()
 
    def forward(self, x, y):        
        
        positive_loss = torch.clamp(torch.log(x), min=-100)
        negative_loss = torch.clamp(torch.log(1-x), min=-100)
        loss = -torch.mean(y * positive_loss + (1-y) * negative_loss)
        return loss
</code></pre>
<p>When I compare the loss to pytorch its BCELoss() I get the same values, the backpropagated gradients however deviate slightly, which degrades my attack performance significantly (if not drastically)</p>
<pre><code class="lang-auto">loss1 = nn.BCELoss()
loss2 = MlaLoss()

input1 = torch.rand(1000, 1000,  requires_grad=True)
input2 = torch.clone(input1).detach()
input2.requires_grad = True
target = torch.ones((1000,1000), requires_grad=False)

cost1 = loss1(input1,target)
cost2 = loss2(input2,target)

cost1.backward()
cost2.backward()

print(torch.sum(cost1 - cost2))
print(torch.sum(input1.grad.data - input2.grad.data))
</code></pre>
<p>The outputs are as follows:</p>
<pre><code class="lang-auto">tensor(0., grad_fn=&lt;SumBackward0&gt;)
tensor(-5.5235e-08)
</code></pre>
<p>How do I fix this deviation? Any suggestions or help would be very much appreciated!</p>
<p>Thanks in advance.</p>

username: mMagmer
cooked: <p>Hi,<br>
I’m not sure, but you should check the backward method of BCE in PyTorch C source.<br>
Here are two of them:</p>
<ul>
<li><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Loss.cpp#L272" class="inline-onebox" rel="noopener nofollow ugc">pytorch/Loss.cpp at master · pytorch/pytorch · GitHub</a></li>
<li><a href="https://github.com/torch/nn/blob/master/lib/THNN/generic/BCECriterion.c#L57" class="inline-onebox" rel="noopener nofollow ugc">nn/BCECriterion.c at master · torch/nn · GitHub</a></li>
</ul>
<p>both of them utilize <code>d(L)/d(x) = -w (y - x) / (x - x^2)</code> as backward method.<br>
But they’re different in using epsilon.<br>
First one uses <code>d(L)/d(x) = -w*(y - x) / max((x - x^2),EPS)</code><br>
and for the second one, derivative is <code>-w*(y - x) / ((1. - x + EPS) * (x + EPS))</code><br>
I go with the first one.<br>
BTW, if you want to use GPU, you should check what the exact implementation is for it.<br>
still I don’t think that this small difference causes your problem. It’s in the floating-point error range.</p>

username: Erwin_van_Thiel
cooked: <p>Thank you for your reply. Im using torch so the second one applies to my case. Then I guess that the backward operations of the operations used in my loss function do not align with the backward used in here <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Loss.cpp#L272" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Loss.cpp#L272</a>. Does the backward of a division also work with an epsilon?</p>

username: mMagmer
cooked: <p>¯\_(ツ)_/¯<br>
Probably yes!</p>