How do I debug a CUDA error from PyTorch?

rreddy78: <p>Hello all,</p>
<p>I am trying to replicate the steps from this repo: <a href="https://github.com/cattaneod/CMRNet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - cattaneod/CMRNet: Code for "CMRNet: Camera to LiDAR-Map Registration" (ITSC 2019) - WIP</a><br>
And I have run into a CUDA error which I dont know how to debug and find the root cause.</p>
<p>CUDA_LAUNCH_BLOCKING=1 python preprocess/kitti_maps.py --sequence 00 --kitti_folder ./KITTI_ODOMETRY/<br>
Sequnce:  00</p>
<p>gives me:<br>
Traceback (most recent call last):<br>
File “preprocess/kitti_maps.py”, line 108, in <br>
local_map = voxelized.detach().clone()<br>
<strong>RuntimeError: CUDA error: <strong>global</strong> function call is not configured</strong></p>
<p>I use:<br>
Ubuntu 20.04<br>
Device 0: “NVIDIA RTX A3000 Laptop GPU”<br>
CUDA Driver Version / Runtime Version 11.6 / 11.6</p>
<p>I use conda environment python 3.7. Have installed CuPy for CUDA 11.6 and then installed the torch and torchvision as given here: <a href="https://discuss.pytorch.org/t/pytorch-cuda-11-6/149647" class="inline-onebox">PyTorch + CUDA 11.6</a></p>
<p>After this I installed the requirements.txt and following steps.</p>
<p>How can one debug such an issue. Am I using too recent pytorch and CUDA that its causing this problem?</p>
<p>Thanks a lot…</p>

ptrblck: <p>The error is raised if no launch parameters were passed to the kernel launch, so I assume a function is running into a previous failure and the actual error message is wrong.<br>
Could you post a minimal, executable code snippet reproducing the issue so that we could try to debug it, please?</p>

rreddy78: <p>Thanks. I will try to make the snippet. In the meantime I have a temporary workaround by computing the same on CPU. There is another problem with actual training now which I am checking</p>

rreddy78: <p>There is an another piece of code that is producing the exact same error. I dont know how to isolate the problem.</p>
<pre>CUDA_LAUNCH_BLOCKING=1 python evaluate_iterative_single_CALIB.py with test_sequence=00 maps_folder=local_maps_0.1 data_folder=./KITTI_ODOMETRY/sequences/ weight="['./checkpoints/iter1.tar','./checkpoints/iter2.tar','./checkpoints/iter3.tar']"
WARNING - CMRNet-evaluate-iterative - No observers have been added to this run
INFO - CMRNet-evaluate-iterative - Running command 'main'
INFO - CMRNet-evaluate-iterative - Started
TEST SET - Not found: ./KITTI_ODOMETRY/sequences/test_RT_seq00_10.00_2.00.csv
Generating a new one
4541
  0%|                                                                                                                                | 0/4541 [00:00&lt;?, ?it/s]Init worker 0 with seed 610730943
Init worker 1 with seed 610730944
Init worker 2 with seed 610730945
Init worker 3 with seed 610730946
Init worker 4 with seed 610730947
Init worker 5 with seed 610730948

ERROR - CMRNet-evaluate-iterative - Failed after 0:00:03!
Traceback (most recent calls WITHOUT Sacred internals):
  File "evaluate_iterative_single_CALIB.py", line 308, in main
    rotated_point_cloud = rotate_forward(point_cloud, RT1)
  File "/home/sxv1kor/Temp/CMRNet/utils.py", line 79, in rotate_forward
    return rotate_points_torch(PC, R, T, inverse=True)
  File "/home/sxv1kor/Temp/CMRNet/utils.py", line 46, in rotate_points_torch
    RT = R.clone()
RuntimeError: CUDA error: __global__ function call is not configured
</pre>
<p>It seems to always happen when .clone() is called. Atleast that is what is common between the the errors.<br>
Any ideas how to isolate the bug in this larger script that I am running ?</p>

rreddy78: <p>I found something:</p>
<p>If I replace clone() with detach() then that error disappears strangely.</p>
<pre>def rotate_points_torch(PC, R, T=None, inverse=True):
    if T is not None:
        R = quat2mat(R)
        T = tvector2mat(T)
        RT = torch.mm(T, R)
    else:
        #RT = R.clone()
         RT = R.detach()
    if inverse:
        RT = RT.inverse()
</pre>
<p>In the code I have commented the RT = R.clone() and replaced with detach(). Now the error is as follows:</p>
<pre>ERROR - CMRNet-evaluate-iterative - Failed after 0:00:03!
Traceback (most recent calls WITHOUT Sacred internals):
  File "evaluate_iterative_single_CALIB.py", line 308, in main
    rotated_point_cloud = rotate_forward(point_cloud, RT1)
  File "/home/sxv1kor/Temp/CMRNet/utils.py", line 80, in rotate_forward
    return rotate_points_torch(PC, R, T, inverse=True)
  File "/home/sxv1kor/Temp/CMRNet/utils.py", line 49, in rotate_points_torch
    RT = RT.inverse()
RuntimeError: CUDA error: __global__ function call is not configured
</pre>
<p>The error does not seem to have much with clone() function but something else…</p>
<p>The calling code is as follows:</p>
<pre> R = quat2mat(target_rot[0])
        T = tvector2mat(target_transl[0])
        RT1_inv = torch.mm(T, R)
        RT1 = RT1_inv.clone().inverse()

        rotated_point_cloud = rotate_forward(point_cloud, RT1)
        RTs = [RT1]

</pre>

ptrblck: <p>Yes, you are delaying the error report as was my previous speculation:</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="152486">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>so I assume a function is running into a previous failure and the actual error message is wrong.</p>
</blockquote>
</aside>
<p>If no proper CUDA error checking is performed the next CUDA operation might be running into the “sticky” error and report the error message, so I think you are right that neither <code>clone()</code> nor <code>inverse</code> are the root cause of the issue but are just reporting “an error” as the CUDA context is corrupt.<br>
Since I don’t know which call causes the issue in the first place (and doesn’t seem to properly report it) I would also not bet that the error message is actually correct.</p>

rreddy78: <p>yes, could be. On the other hand, what I found with the previous (first)  example of preprocess/kitti_maps.py, is that<br>
if I eliminate the calls to .clone() method as given below, then the script runs fine on GPU (atleast On a Ubuntu 18.04 with CUDA 11.3 + matched PyTorch).</p>
<p>Here I give a diff of the changes I did:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50ff52110ca8586d2f1d0172572e7dd774511659.png" data-download-href="https://discuss.pytorch.org/uploads/default/50ff52110ca8586d2f1d0172572e7dd774511659" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50ff52110ca8586d2f1d0172572e7dd774511659_2_690x400.png" alt="image" data-base62-sha1="byxdBwH8v5Kuvn8TiVzf1x7fG4x" width="690" height="400" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50ff52110ca8586d2f1d0172572e7dd774511659_2_690x400.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50ff52110ca8586d2f1d0172572e7dd774511659_2_1035x600.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/0/50ff52110ca8586d2f1d0172572e7dd774511659.png 2x" data-dominant-color="39132A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1255×729 114 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The same code on a Ubuntu 20.04 with CUDA 11.6 and matched PyTorch gives me error:</p>
<pre>python preprocess/kitti_maps.py  --sequence 00 --map map-00_0.1_0-4541.pcd --kitti_folder ./KITTI_ODOMETRY/
Traceback (most recent call last):
  File "preprocess/kitti_maps.py", line 110, in &lt;module&gt;
    local_map = torch.mm(pose, voxelized).t()
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)`
</pre>

ptrblck: <p>Try to come up with a minimal code snippet to reproduce the issue, as right now it seems that random operations are running into the error. I.e. even the cublas error could be caused by just running into an already corrupted CUDA context.</p>

rreddy78: <p>Its runs  with the following environment:</p>
<ol>
<li>Python 3.6 Conda Env</li>
<li>GPU with Turing architecture</li>
<li>CUDA 10.2 toolkit</li>
<li>PyTorch installed with command:<br>
conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch</li>
</ol>
<p>And also a matching cudnn library as specified in the pytorch</p>