RuntimeError: CUDA error: an illegal memory access was encountered (launch_kernel at C:/w/b/windows/pytorch/aten/src\ATen/native/cuda/CUDALoops.cuh:217)

username: Guimin_Hu
cooked: <p>I’m a student.<br>
I gets errors when my model runs to code ‘loss.backward()’. I display the details of error.</p>
<p>ERROR 1:<br>
File “F:/CODE/BHC/main.py”, line 175, in <br>
p, r, f1 = main(configs, fold_id)<br>
File “F:/CODE/BHC/main.py”, line 73, in main<br>
loss.backward()<br>
File “F:\Common_V\lib\site-packages\torch\tensor.py”, line 198, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “F:\Common_V\lib\site-packages\torch\autograd_<em>init</em>_.py”, line 100, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: CUDA error: an illegal memory access was encountered (launch_kernel at C:/w/b/windows/pytorch/aten/src\ATen/native/cuda/CUDALoops.cuh:217)<br>
(no backtrace available)</p>
<p>ERROR 2:<br>
Traceback (most recent call last):<br>
File “F:/CODE/BHC/main.py”, line 175, in <br>
p, r, f1 = main(configs, fold_id)<br>
File “F:/CODE/BHC/main.py”, line 73, in main<br>
loss.backward()<br>
File “F:\Common_V\lib\site-packages\torch\tensor.py”, line 198, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “F:\Common_V\lib\site-packages\torch\autograd_<em>init</em>_.py”, line 100, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling <code>cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)</code> (gemm at …\aten\src\ATen\cuda\CUDABlas.cpp:165)<br>
(no backtrace available)</p>
<p>I set breakpoint at the line of ‘loss.backward()’  and find a problem that may led to occurrence of errors,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/4/246d16ce025c2ed474d3edf9f61947ac7b893d3d.png" data-download-href="https://discuss.pytorch.org/uploads/default/246d16ce025c2ed474d3edf9f61947ac7b893d3d" title="捕获111"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/246d16ce025c2ed474d3edf9f61947ac7b893d3d_2_690x244.png" alt="捕获111" data-base62-sha1="5ceUuHR0TADA2by85ZvlFXuxzXL" width="690" height="244" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/246d16ce025c2ed474d3edf9f61947ac7b893d3d_2_690x244.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/246d16ce025c2ed474d3edf9f61947ac7b893d3d_2_1035x366.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/4/246d16ce025c2ed474d3edf9f61947ac7b893d3d.png 2x" data-dominant-color="F2F5F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">捕获111</span><span class="informations">1314×466 22.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
but i don’t know the reason.</p>
<p>additionally, i try to some ways to fix this bug, such as, set ‘torch.cuda.set_device(0)’ , set ‘torch.backends.cudnn.enabled=False’, or decrease the batch size, the batch size is one.<br>
but all these ways don’t work.</p>
<p>I’m confused and sad, I hope somebody could help me. I will be very appreciated for your reply.</p>

username: ptrblck
cooked: <p>Could you rerun the script via:</p>
<pre><code class="lang-auto">CUDA_LAUNCH_BLOCKING=1 python script.py args
</code></pre>
<p>and post the complete stack trace here?<br>
Also, which GPU, PyTorch, CUDA, and cudnn versions are you using?</p>