Exception in BCECriterion.cu:42

cherepanovic: <p>Is someone aware of this exception? Raised after 70th epoch (it does not depend on epoch… did it again and got the exception after 40th epoch)</p>
<pre><code class="lang-auto">Train Epoch: 70 [0/6742 (0%)]   Loss: -431231.800000
/opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THCUNN/BCECriterion.cu:42: Acctype bce_functor&lt;Dtype, Acctype&gt;::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references&lt;thrust::device_reference&lt;float&gt;, thrust::device_reference&lt;float&gt;, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type&gt;, Dtype = float, Acctype = float]: block: [4,0,0], thread: [224,0,0] Assertion `input &gt;= 0. &amp;&amp; input &lt;= 1.` failed.
</code></pre>
<p>Thanks</p>

kshitij: <p>Looks like at some point in training the model is outputting value outside the range of 0 to 1. Are you sure that the predicted value from the model are in range 0 to 1?</p>
<p>Can you put a <code>try-except</code> to see the value of the model prediction when Error is raised?</p>

cherepanovic: <p>I used the MNIST dataset. The values are between 0 and 255 (included).</p>
<p>the predicted values have such values (a snippet)</p>
<pre><code>    0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,
    5.4309e-15, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,
    0.0000e+00, 7.7825e-17, 1.0000e+00, 1.0000e+00, 1.0000e+00, 1.0000e+00,
    1.0000e+00, 1.0000e+00, 1.0000e+00, 1.0000e+00, 1.0000e+00, 1.0000e+00,
    1.0000e+00, 1.0000e+00, 1.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,
    0.0000e+00, 2.2867e-38, 1.7363e-37, 9.8193e-22, 9.3205e-27, 0.0000e+00,</code></pre>

cherepanovic: <p>you are right my loss function is the F.binary_cross_entropy.</p>

kshitij: <p>The predicted value for the model should be between 0 and 1.</p>
<pre><code class="lang-python">import torch
import torch.nn.functional as F
preds = torch.ones(3) + 0.01
target = torch.ones(3)
F.binary_cross_entropy(preds, target) # Crashes as preds is not between 0 and 1.
</code></pre>
<p>I suspect this is what is happening in the epoch in which it is crashing i.e. preds is not in range 0 and 1</p>
<p>For the epoch it crashes can you do,</p>
<pre><code class="lang-python">try:
    # training step
except:
    mask = (preds &gt; 1) + (preds &lt; 0) # see if any of the value in mask is True
    print(mask.sum()) # If non-zero, then predictions are out of the range.
</code></pre>
<p>Most probably you have done this but do verify if you have applied sigmoid before passing the preds to the model.</p>
<p>Or you can use <a href="https://pytorch.org/docs/stable/nn.functional.html#binary-cross-entropy-with-logits" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.functional.html#binary-cross-entropy-with-logits</a></p>
<p>Hope this helps.</p>

cherepanovic: <blockquote>
<p>Or you can use <a href="https://pytorch.org/docs/stable/nn.functional.html#binary-cross-entropy-with-logits" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional — PyTorch 2.1 documentation</a></p>
</blockquote>
<p>did not help, loss function returened a nan</p>
<p><code>tensor(nan, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;)</code></p>

cherepanovic: <blockquote>
<p>F.binary_cross_entropy(preds, target) # Crashes as preds is not between 0 and 1.</p>
</blockquote>
<p>not the same crash: BCECriterion.c:62 (in my case it was BCECriterion.cu:42)</p>
<p>but both of them reveal the wrong input.</p>
<p>thank you!</p>

cherepanovic: <p>the funny fact is that the last layer of my model is already a sigmoid function</p>
<pre><code class="lang-auto">
return F.sigmoid(self.fc6(h))
</code></pre>
<p>that means that the results should be within the required range</p>

kshitij: <p>Surprising to see the error.</p>

cherepanovic: <p>(btw) I tried the same execution with smaller learning rates 1e-4/1e-5/1e-6 over 150 iterations and didn’t get any errors.</p>
<p>Still waiting for help regarding this issue.</p>
<p>(my replicated post in github <a href="https://github.com/pytorch/pytorch/issues/36647" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/36647</a>)</p>