BCELoss Backward When The Input is Zero

NoobCoder: <p>Hello,</p>
<p>I want to check if my model is calculating the gradient ok or not.</p>
<p>I have used nn.BCELoss for the loss function. Based on the Pytorch Website, it is mentioned that the output of the loss will be clipped to -100 if it is smaller than this value.</p>
<p>My question is that what will happen in the backward? If the <code>y</code> is zero, because of the gradient it will be in denominator, and the gradient would be <code>inf</code>. There is no explanation about the backward in the site. Can you explain how should I calculate it? Should I add it with <code>epsilon</code>? or …</p>
<p>Also, I have seen a way for getting the gradient of loss respect to the prediction is using <code>hook</code>, but there are small examples and I do not know how should I do that? Can someone say what is the procedure?</p>
<p>Thanks,</p>

KFrank: <p>Hi Amirali!</p>
<aside class="quote no-group" data-username="NoobCoder" data-post="1" data-topic="84580" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e9bcb4/48.png" class="avatar"> NoobCoder:</div>
<blockquote>
<p>I have used nn.BCELoss for the loss function. Based on the Pytorch Website, it is mentioned that the output of the loss will be clipped to -100 if it is smaller than this value.</p>
</blockquote>
</aside>
<p>Just to be clear, the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCELoss" rel="noopener nofollow ugc">BCELoss</a> documentation says that the individual<br>
<code>log()</code> terms in <code>BCELoss</code> will be clamped to be <code>&gt;= -100</code>, not the loss<br>
itself.</p>
<aside class="quote no-group">
<blockquote>
<p>My question is that what will happen in the backward? If the <code>y</code> is zero, because of the gradient it will be in denominator, and the gradient would be <code>inf</code>.</p>
</blockquote>
</aside>
<p>My understanding of this (but I haven’t checked the code) is that<br>
pytorch uses the “clamped-log” version of <code>BCELoss</code> rather than the<br>
“true” <code>BCELoss</code>, and that the gradient produced by <code>backward()</code><br>
is the gradient of this modified loss function.  That is, you modify<br>
the loss function and then take the modified function’s gradient,<br>
rather than modifying the loss function and its gradient separately.<br>
(The derivative of the clamped-log term is well defined, and diverges<br>
nowhere.  It is zero in the clamped regime, but this doesn’t cause<br>
any divergence.)</p>
<p>(As an aside, the documentation uses <code>x</code> for the <code>input</code> (prediction)<br>
and <code>y</code> for the target.  It is <code>x</code> that appears in the <code>log()</code> terms, so it is<br>
<code>log (x)</code> that gets clamped.  <code>backward()</code> also computes the gradient<br>
with respect to <code>x</code> (not <code>y</code>).)</p>
<p>Best.</p>
<p>K. Frank</p>

NoobCoder: <p>Hello Frank,</p>
<p>Thanks for your answer. Yes, you are right about the <code>clamp-log</code>.I have implemented it, and I am getting same error values. Just for someone need to check, this is the code:</p>
<pre><code class="lang-auto">def BCE_Loss(x, y, weight):

    clamp_log_x = np.log(x)
    clamp_log_x[clamp_log_x &lt;-100] = -100

    clamp_log_1_x = np.log(1-x)
    clamp_log_1_x[clamp_log_1_x &lt;-100] = -100

    cal_loss = np.multiply(y, clamp_log_x) + np.multiply(1-y, clamp_log_1_x)
    cal_loss = np.multiply(cal_loss, -1*weight).sum()

    return cal_loss
</code></pre>
<p>The above function, calculate the BCE loss with weighting.</p>

NoobCoder: <p>I implemented the gradient of the loss.<br>
I compared my gradient to layers with Pytorch’s one, and because it was different, I used ‘Autograd’ to write a model with my own backward.</p>
<pre><code class="lang-auto">@staticmethod
    def backward(ctx, grad_output):

        print("grad:", grad_output.shape)
        print(grad_output)
</code></pre>
<p>Before this model, I did not know how to access the gradient of the loss to the output of my previous model, so I had to calculate the weights of model and compare those to that. Now, with the above function, I was able to access the gradient of loss, so I tried to compare to mine. Interesting thing is that these two values are not equal.</p>
<p>My loss is weighting BCEloss with these code:</p>
<pre><code class="lang-auto">criterion = nn.BCELoss(reduction='none')
loss = criterion(predicted, heatmaps)
weight_loss = loss * weights
sum_loss = torch.sum(weight_loss, dim=1)
avg_loss = torch.mean(sum_loss)
</code></pre>
<p>and in the backward, I just call:</p>
<pre><code class="lang-auto">avg_loss.backward()
</code></pre>
<p>I am computing the gradient of BCEloss with this function:</p>
<pre><code class="lang-auto">def grad_BCEloss(x, y, weight):

    temp_x = np.array([1/num if np.log(num) &gt;=-100 else 0 for num in temp_x]])
    temp_1_x = np.array([1/num if np.log(num) &gt;=-100 else 0 for num in 1 - temp_x])

    dloss_dx = weight*(-np.multiply(y, temp_x) + np.multiply(1-y, temp_1_x))
    
    return dloss_dx
</code></pre>
<p>The strange thing is that the value of my function with Pytorch’s one is not the same. There are input values:</p>
<pre><code class="lang-auto">weights: tensor([[0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033,
         0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033,
         0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033,
         0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033,
         0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033,
         0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033,
         0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033,
         0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033,
         0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033,
         0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033,
         0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033,
         0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033,
         0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033, 0.0033,
         0.0033, 0.0033, 0.0033, 0.0033]])


X: tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0674,
         0.0766, 0.0763, 0.0668, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0769, 0.1063, 0.1312, 0.1304, 0.1048, 0.0755, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0674, 0.1063, 0.1725, 0.2489, 0.2460, 0.1684, 0.1038, 0.0659,
         0.0000, 0.0000, 0.0000, 0.0766, 0.1312, 0.2489, 0.5474, 0.4377, 0.2405,
         0.1273, 0.0747, 0.0000, 0.0000, 0.0000, 0.0763, 0.1304, 0.2460, 0.4377,
         0.4289, 0.2378, 0.1265, 0.0744, 0.0000, 0.0000, 0.0000, 0.0668, 0.1048,
         0.1684, 0.2405, 0.2378, 0.1645, 0.1023, 0.0653, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0755, 0.1038, 0.1273, 0.1265, 0.1023, 0.0742, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0659, 0.0747, 0.0744, 0.0653, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000]], grad_fn=&lt;ViewBackward&gt;)


Y: tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0.]])
</code></pre>
<p>Which each of are tensor of size [batch_size, 121, 1]. This my output:</p>
<pre><code class="lang-auto">dloss_dy: [[0.00334308 0.00334308 0.00334308 0.00334308 0.00334308 0.00334308
  0.00334308 0.00334308 0.00334308 0.00334308 0.00334308 0.00334308
  0.00334308 0.00334308 0.00334308 0.00334308 0.00334308 0.00334308
  0.00334308 0.00334308 0.00334308 0.00334308 0.00334308 0.00334308
  0.00334308 0.00334308 0.00358469 0.00362035 0.00361928 0.00358226
  0.00334308 0.00334308 0.00334308 0.00334308 0.00334308 0.00334308
  0.00362143 0.00374088 0.00384777 0.00384422 0.00373434 0.0036161
  0.00334308 0.00334308 0.00334308 0.00334308 0.00358469 0.00374088
  0.00403994 0.00445096 0.00443402 0.0040201  0.00373009 0.00357909
  0.00334308 0.00334308 0.00334308 0.00362035 0.00384777 0.00445096
  0.00738696 0.00594555 0.00440164 0.00383053 0.00361301 0.00334308
  0.00334308 0.00334308 0.00361928 0.00384422 0.00443402 0.00594555
  0.00585394 0.00438616 0.00382722 0.00361199 0.00334308 0.00334308
  0.00334308 0.00358226 0.00373434 0.0040201  0.00440164 0.00438616
  0.00400135 0.00372389 0.00357677 0.00334308 0.00334308 0.00334308
  0.00334308 0.0036161  0.00373009 0.00383053 0.00382722 0.00372389
  0.00361098 0.00334308 0.00334308 0.00334308 0.00334308 0.00334308
  0.00334308 0.00357909 0.00361301 0.00361199 0.00357677 0.00334308
  0.00334308 0.00334308 0.00334308 0.00334308 0.00334308 0.00334308
  0.00334308 0.00334308 0.00334308 0.00334308 0.00334308 0.00334308
  0.00334308]]
</code></pre>
<p>Which makes sense that when <code>x</code> and <code>y</code> are <code>0</code>, the gradient is the weighting, but this is Pytorch’s one:</p>
<pre><code class="lang-auto">tensor([[[-0.0000, -0.0000, -0.0000, -0.0000, -0.0000, -0.0000, -0.0000, -0.0000,
          -0.0000, -0.0000, -0.0000],
         [-0.0000, -0.0000, -0.0000, -0.0000, -0.0000, -0.0000, -0.0000, -0.0000,
          -0.0000, -0.0000, -0.0000],
         [-0.0000, -0.0000, -0.0000, -0.0000, 0.0036, 0.0036, 0.0036, 0.0036,
          -0.0000, -0.0000, -0.0000],
         [-0.0000, -0.0000, -0.0000, 0.0036, 0.0037, 0.0038, 0.0038, 0.0037,
          0.0036, -0.0000, -0.0000],
         [-0.0000, -0.0000, 0.0036, 0.0037, 0.0040, 0.0045, 0.0044, 0.0040,
          0.0037, 0.0036, -0.0000],
         [-0.0000, -0.0000, 0.0036, 0.0038, 0.0045, 0.0074, 0.0059, 0.0044,
          0.0038, 0.0036, -0.0000],
         [-0.0000, -0.0000, 0.0036, 0.0038, 0.0044, 0.0059, 0.0059, 0.0044,
          0.0038, 0.0036, -0.0000],
         [-0.0000, -0.0000, 0.0036, 0.0037, 0.0040, 0.0044, 0.0044, 0.0040,
          0.0037, 0.0036, -0.0000],
         [-0.0000, -0.0000, -0.0000, 0.0036, 0.0037, 0.0038, 0.0038, 0.0037,
          0.0036, -0.0000, -0.0000],
         [-0.0000, -0.0000, -0.0000, -0.0000, 0.0036, 0.0036, 0.0036, 0.0036,
          -0.0000, -0.0000, -0.0000],
         [-0.0000, -0.0000, -0.0000, -0.0000, -0.0000, -0.0000, -0.0000, -0.0000,
          -0.0000, -0.0000, -0.0000]]])
</code></pre>
<p>A strange thing happen:</p>
<ol>
<li>Where the <code>x</code> is not <code>0</code>, they have same value. (approximately)</li>
<li>Where the <code>x</code> is <code>0</code>, it is zero. (the first term because of <code>clamp-log</code> is zero, but the second term is (1-y)/(1-x) --&gt; 1/1, so it would be multiply by weights, so it should have the value of weights.</li>
</ol>
<p>Can anyone explain why is this happening?</p>

NoobCoder: <p>Also, for <a href="https://discuss.pytorch.org/t/bceloss-backward-when-the-input-is-zero/84580/3" class="inline-onebox">BCELoss Backward When The Input is Zero</a>, this function, I printed more values, when the total loss in below <code>1</code>, I have same value with <code>nn.BCELoss</code>, but in values more than <code>1</code>,  I have a lot of difference. For example, with these values:</p>
<pre><code class="lang-auto">x: tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0587, 0.0732,
         0.0820, 0.0792, 0.0668, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0587,
         0.0850, 0.1194, 0.1449, 0.1363, 0.1033, 0.0718, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0732, 0.1194, 0.2001, 0.2839, 0.2528, 0.1588, 0.0949, 0.0599,
         0.0000, 0.0000, 0.0000, 0.0820, 0.1449, 0.2839, 0.5244, 0.4029, 0.2074,
         0.1103, 0.0657, 0.0000, 0.0000, 0.0000, 0.0792, 0.1363, 0.2528, 0.4029,
         0.3431, 0.1903, 0.1052, 0.0639, 0.0000, 0.0000, 0.0000, 0.0668, 0.1033,
         0.1588, 0.2074, 0.1903, 0.1317, 0.0845, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0718, 0.0949, 0.1103, 0.1052, 0.0845, 0.0622, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0599, 0.0657, 0.0639, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000]], grad_fn=&lt;ViewBackward&gt;)


y: tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1667, 0.2857, 0.5000, 0.6667,
         0.5000, 0.2857, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1818, 0.3333,
         0.6667, 1.0000, 0.6667, 0.3333, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.1667, 0.2857, 0.5000, 0.6667, 0.5000, 0.2857, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.1333, 0.2000, 0.2857, 0.3333, 0.2857, 0.2000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1333, 0.1667, 0.1818, 0.1667,
         0.1333, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.0000, 0.0000]])
</code></pre>
<p>The nn.BCELoss is <code>118.65484619140625</code>, but with my function it is <code>424.10596</code>. I do not know why in some places I get the right answer and somewhere not.</p>
<p>Can you help on this ? <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/alband">@albanD</a></p>

ptrblck: <p>I’m not completely sure, what you are trying to achieve, but you might saturate the sigmoid function with high or low values as shown in <a href="https://discuss.pytorch.org/t/difference-between-bceloss-and-bcewithlogitsloss-when-using-large-values/54053/2">this post</a>.</p>

NoobCoder: <p>Thanks for your reply.</p>
<p>The reason I am doing all this is that after my model’s output, I have done some changes. The Pytorch is calculating the gradient, but the problem is that I have implemented that changes in another way and the value of gradient is different. (with the same initialization, same input and … ) Based on these, I think the Pytorch cannot calculate the gradient correctly, so I used <code>autograd</code> to define my backward too.</p>
<p>I am trying to check the BCELoss’s gradient and output. I have used sigmoid before these layers, but I do not think the problem is Sigmoid here.</p>
<p>I have <code>x</code> and <code>y</code>, and I am trying to compute the loss value and it’s gradient.</p>
<p>The loss value is same in 99% times, but the gradient is never same. I have out the code and the values that I am not getting in <a href="https://discuss.pytorch.org/t/bceloss-backward-when-the-input-is-zero/84580/4">here</a>.</p>
<p>In Pytorch implementation, is <code>clamp-log</code> is used for backward too? I mean is this statement right?</p>
<blockquote>
<p>if the <code>log(x)</code> is smaller than <code>-100</code>, the gradient is <code>zero</code><br>
if the <code>log(x)</code> is bigger or equal than <code>-100</code>, the gradient is <code>1/x</code></p>
</blockquote>
<p>If yes, why I am not getting the same gradient to compare to Pytorch?</p>