Bce loss when prediction is nearly to 1

Wjk666: <p>Hi, I wish to use <code>bceloss</code> to calculate the prediction loss. But at the beginning of the training, the prediction is nearly about 1. Then as for the bceloss, it occurs some error. Looking forward to your help!<br>
For a toy example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
a = torch.randn(512,4)
leakyrelu = nn.LeakyReLU(0.2)
att = nn.Softmax(dim=-1)(-leakyrelu(a))
b = torch.ones(512, 14541, 4)
score = torch.einsum('bnk,bk-&gt;bn', [b, att])
bce = nn.BCELoss()
label = torch.full((512, 14541), 3.141592)
bce(score, label)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9df658de282a0d94a10fe0814ee08710623bfe7c.png" alt="image" data-base62-sha1="mxoL4ghXRCHY4Gf0Aj1Y0lXGxxW" width="436" height="308"><br>
errors are following:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-18-d7e1c60c672a&gt; in &lt;module&gt;()
      8 bce = nn.BCELoss()
      9 label = torch.full((512, 14541), 3.141592)
---&gt; 10 bce(score, label)

2 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in binary_cross_entropy(input, target, weight, size_average, reduce, reduction)
   2760         weight = weight.expand(new_size)
   2761 
-&gt; 2762     return torch._C._nn.binary_cross_entropy(input, target, weight, reduction_enum)
   2763 
   2764 

RuntimeError: all elements of input should be between 0 and 1
</code></pre>
<p>Actually, in my empirical experiments, the error is <code>Reduce failed to synchronize: device-side assert triggered</code>. Then I make a toy experiment above, it also seems strange.</p>

KFrank: <p>Hi Wjk!</p>
<p>The short answer is that <code>score.max() &gt; 1.0</code>.</p>
<aside class="quote no-group quote-modified" data-username="Wjk666" data-post="1" data-topic="119289" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/e8c25b/48.png" class="avatar"> Wjk666:</div>
<blockquote>
<p>Then as for the bceloss, it occurs some error.</p>
<pre><code class="lang-auto">att = nn.Softmax(dim=-1)(-leakyrelu(a))
</code></pre>
</blockquote>
</aside>
<p>Because you calculate <code>Softmax()</code>, the last dimension of <code>att</code> will<br>
sum to one, <em>up to floating-point round-off error.</em></p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">b = torch.ones(512, 14541, 4)
score = torch.einsum('bnk,bk-&gt;bn', [b, att])
</code></pre>
</blockquote>
</aside>
<p>You “contract” the last dimension of <code>att</code> with <code>torch.ones()</code>.  That is,<br>
you multiply the elements of <code>att</code> by <code>1.0</code> and then sum along the<br>
last dimension.  Therefore the elements of <code>score</code> are all <code>1.0</code>, <em>up<br>
to floating-point round-off error.</em></p>
<p>By default, pytorch prints out tensor values to 5 digits, so when<br>
you print out <code>score.max()</code> you only see <code>1.0000</code>.  Print out<br>
<code>score.max() - 1.0</code>, and you will see a small number that is<br>
slightly greater than zero.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">bce = nn.BCELoss()
label = torch.full((512, 14541), 3.141592)
bce(score, label)
</code></pre>
</blockquote>
</aside>
<p>Here, you pass into <code>BCELoss</code> values that are (slightly, due to round-off<br>
error) greater than one, so <code>BCELoss</code> raises the exception you posted:</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">RuntimeError: all elements of input should be between 0 and 1
</code></pre>
</blockquote>
</aside>
<p>Best.</p>
<p>K. Frank</p>

Wjk666: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="119289">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Because you calculate <code>Softmax()</code>, the last dimension of <code>att</code> will<br>
sum to one, <em>up to floating-point round-off error.</em></p>
</blockquote>
</aside>
<p>So I need to control it before bceloss, is it right? Such as</p>
<pre><code class="lang-auto">score[score &gt; 1.0] = 1.0
</code></pre>

KFrank: <p>Hi Wjk!</p>
<aside class="quote no-group" data-username="Wjk666" data-post="3" data-topic="119289" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/e8c25b/48.png" class="avatar"> Wjk666:</div>
<blockquote>
<p>So I need to control it before bceloss, is it right? Such as</p>
<pre><code class="lang-auto">score[score &gt; 1.0] = 1.0
</code></pre>
</blockquote>
</aside>
<p>Yes, this is a sensible thing to do.</p>
<p>This could also be an appropriate use case for <a href="https://pytorch.org/docs/1.8.1/generated/torch.clamp.html#torch.clamp" rel="noopener nofollow ugc">torch.clamp()</a>.</p>
<p>Note, however, you should only be doing this if the result you are forcing<br>
to be <code>&lt;= 1.0</code> should already be mathematically no greater than 1.0, and<br>
only exceed 1.0 because of round-off error. (as in your toy example).  If in<br>
your real use case the values you use as <code>input</code> for <code>BCELoss</code> could,<br>
absent round-off error, be greater than 1.0, then you should fix that issue.<br>
Simply clamping them to be no greater than 1.0 would just be sweeping<br>
the real problem under the rug.</p>
<p>Best.</p>
<p>K. Frank</p>

Wjk666: <p>Thanks a lot. I have thought clearly.<br>
Best,<br>
Json</p>