RuntimeError CUDA error: device-side assert triggered

shirui-japina: <p>I’m training my model, and <strong>at the third epoch</strong>, got <strong>RuntimeError CUDA error: device-side assert triggered</strong>.<br>
Here is my code, where occurred the error.</p>
<pre><code class="lang-auto"># epoch loop
    for i, (batch_z_16, batch_z_32, batch_z_48, batch_label) in enumerate(data_loader_validation):
        batch_z_16 = batch_z_16.to(device=device, dtype=torch.float) # Exception here
</code></pre>
<p>And I also got message in terminal (using VS Code) as below:</p>
<pre><code class="lang-auto">C:/w/1/s/tmp_conda_3.6_035809/conda/conda-bld/pytorch_1556683229598/work/aten/src/THCUNN/BCECriterion.cu:57: block: [0,0,0], thread: [7,0,0] Assertion `*input &gt;= 0. &amp;&amp; *input &lt;= 1.` failed.
C:/w/1/s/tmp_conda_3.6_035809/conda/conda-bld/pytorch_1556683229598/work/aten/src/THCUNN/BCECriterion.cu:57: block: [0,0,0], thread: [12,0,0] Assertion `*input &gt;= 0. &amp;&amp; *input &lt;= 1.` failed.
C:/w/1/s/tmp_conda_3.6_035809/conda/conda-bld/pytorch_1556683229598/work/aten/src/THCUNN/BCECriterion.cu:57: block: [0,0,0], thread: [15,0,0] Assertion `*input &gt;= 0. &amp;&amp; *input &lt;= 1.` failed.
C:/w/1/s/tmp_conda_3.6_035809/conda/conda-bld/pytorch_1556683229598/work/aten/src/THCUNN/BCECriterion.cu:57: block: [0,0,0], thread: [27,0,0] Assertion `*input &gt;= 0. &amp;&amp; *input &lt;= 1.` failed.
C:/w/1/s/tmp_conda_3.6_035809/conda/conda-bld/pytorch_1556683229598/work/aten/src/THCUNN/BCECriterion.cu:57: block: [0,0,0], thread: [30,0,0] Assertion `*input &gt;= 0. &amp;&amp; *input &lt;= 1.` failed.
</code></pre>
<p>My question is:</p>
<ul>
<li>
<p>Why it doesn’t occur at the same epoch or why it doesn’t occur at the first epoch?<br>
(I tried some times and got that it occurs at different epoch time, even sometimes it doesn’t occur.)</p>
</li>
<li>
<p>It is <strong>device-side</strong> error, so can I fix it as a user? (what is <strong>device-side</strong>?)</p>
</li>
<li>
<p>What exactly it is? problem of CUDA, PyTorch or my Python coding?</p>
</li>
<li>
<p>How can I make it never happens?</p>
</li>
</ul>

albanD: <p>Hi,</p>
<ul>
<li>It looks like an error because some index in you BCECriterion are wrong. This may not happen in all epochs if you drop the last partial batches for example. Or if you generate these inputs on the fly and some of them are wrong.</li>
<li>It’s a device side that says that what you gave as input does not verify some condition. So yes you can fix it by giving correct inputs to that function <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">
</li>
<li>This kind of error is raised when a cuda kernel detect a problem. CUDA is asynchronous, so unless you start your code with <code>CUDA_LAUNCH_BLOCKING=1</code> environment variable, the python stack trace is wrong. Unfortunaly, because of how cuda works, we can’t make these errors much more user-friendly.</li>
<li>The error is in the file <code>BCECriterion.cu</code> so it comes from a call to BCELoss (or a criterion that uses it) You can see that <code>*input &gt;= 0. &amp;&amp; *input &lt;= 1.</code> You might want to check that your values are in [0, 1] as expected from the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCELoss" rel="nofollow noopener">doc</a>.</li>
</ul>