RuntimeError: CUDA error: device-side assert triggered

Peter_Ham: <p>I’m putting my code here:</p>
<pre><code>with torch.no_grad():
        retrieval_one_hot = torch.zeros(k, 10).cuda()
        for batch_idx, (inputs, targets, indexes) in enumerate(testloader):
             inputs, targets, indexes = inputs.to(device), targets.to(device), indexes.to(device)
             net.to(device)
             targets = targets.cuda()#async=True)
             batchSize = inputs.size(0)
             features = net(inputs)
             zz = torch.zeros(batchSize * k, 10).cuda()
             ....
</code></pre>
<p>and it returns the error at the line zz = torch.zeros(batchSize * k, 10).cuda(), with the error message “RuntimeError: CUDA error: device-side assert triggered”, any suggestions?</p>

Tony-Y: <p>Please the error message!</p>

Tony-Y: <aside class="quote" data-post="2" data-topic="17442">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-cuda-runtime-error-59-device-side-assert-triggered-at-pytorch-torch-lib-thc-generic-thcstorage-c-36/17442/2">RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/torch/lib/THC/generic/THCStorage.c:36</a> 
  </div>
  <blockquote>
    Hi, 
First thing is to try to run the code on CPU. CPU code has more checks so it will possibly return a better error message. 
If the CPU code runs without error, then run the same thing with CUDA_LAUNCH_BLOCKING=1 to get a proper error message and stack trace.
  </blockquote>
</aside>

<p>This suggestion might be the case for your problem.</p>

Peter_Ham: <p>The code runs fine on cpu, but fails to run on gpu.</p>

Tony-Y: <p><a href="https://lernapparat.de/debug-device-assert/" rel="nofollow noopener">Debugging CUDA device-side assert in PyTorch</a><br>
This article would help you to debug your code. If you get a better traceback setting <code>CUDA_LAUNCH_BLOCKING=1</code>, post it.</p>

Gaurav_Koradiya: <p>I am also getting error in CPU</p>

Tony-Y: <p>I think you unintentionally used CUDA tensors. Please place every tensor <code>x</code> in CPU memory:</p>
<pre><code class="lang-auto">x = x.cpu()
</code></pre>

christopherney: <p>You should read this post: <a href="https://towardsdatascience.com/cuda-error-device-side-assert-triggered-c6ae1c8fa4c3" rel="nofollow noopener">https://towardsdatascience.com/cuda-error-device-side-assert-triggered-c6ae1c8fa4c3</a></p>

jh_chen: <p>The reason why you met this problem becasue your “target” labels  include minus values such as “-1”.<br>
I find this reason when i met this problem. hope useful!</p>

Yufeng_Zhang: <p>Very helpful! Thanks a lot!</p>

Baltz: <p>In my case, this error is caused because my loss function just receive values between [0, 1], and i was passing other values.</p>
<p>So, normalizing my loss function input, solve this:</p>
<pre><code class="lang-auto">    saida_G -= saida_G.min(1, keepdim=True)[0]
    saida_G /= saida_G.max(1, keepdim=True)[0]
</code></pre>
<p>Read this: <a href="https://towardsdatascience.com/cuda-error-device-side-assert-triggered-c6ae1c8fa4c3" rel="noopener nofollow ugc">link</a></p>

SaharCarmel: <p>Another issue that might raise this error is mismatch with the last layer of the network. Validate that the output of the network is the same as the number of labels.</p>

Yanting_Huang: <p>Yes, the same situation caused this error for me. I just fixed that. Thanks!</p>

milan_kalkenings: <p>for noobs like me:<br>
use</p>
<pre><code class="lang-auto">import os
os.environ["CUDA_LAUNCH_BLOCKING"] = "1"
</code></pre>
<p>on top of your script and run the script to get a way better stack trace. in my case  the cuda error told me to look at a completely different line which caused an hour of debugging</p>