Cuda error when convert tensor to cpu

Haron_Wan: <p>I got this error while running GAT, can anyone help me with it? Thanks a lot.</p>
<p>File “/home/disk1/users/MyGat/MyGat0417/utils.py”, line 114, in _parse_and_check_input<br>
y_true = y_true.detach().cpu().numpy()</p>
<p>RuntimeError: CUDA error: device-side assert triggered<br>
terminate called after throwing an instance of ‘dmlc::Error’<br>
what():  [22:21:45] /opt/dgl/src/runtime/cuda/cuda_device_api.cc:103: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: CUDA: device-side assert triggered<br>
Stack trace:<br>
[bt] (0) /home/disk1/users/.local/lib/python3.6/site-packages/dgl/libdgl.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x4f) [0x7f13167772cf]<br>
[bt] (1) /home/disk1/guoweifeng/.local/lib/python3.6/site-packages/dgl/libdgl.so(dgl::runtime::CUDADeviceAPI::FreeDataSpace(DLContext, void*)+0x15c) [0x7f1316fc44ec]<br>
[bt] (2) /home/disk1/users/.local/lib/python3.6/site-packages/dgl/libdgl.so(dgl::runtime::NDArray::Internal::DefaultDeleter(dgl::runtime::NDArray::Container*)+0x1ad) [0x7f1316e824fd]<br>
[bt] (3) /home/disk1/users/.local/lib/python3.6/site-packages/dgl/libdgl.so(dgl::UnitGraph::COO::~COO()+0x127) [0x7f1316f9ea37]<br>
[bt] (4) /home/disk1/users/.local/lib/python3.6/site-packages/dgl/libdgl.so(dgl::UnitGraph::~UnitGraph()+0x1ba) [0x7f1316f9e7ca]<br>
[bt] (5) /home/disk1/users/.local/lib/python3.6/site-packages/dgl/libdgl.so(dgl::HeteroGraph::~HeteroGraph()+0x119) [0x7f1316e9b4e9]<br>
[bt] (6) /home/disk1/users/.local/lib/python3.6/site-packages/dgl/libdgl.so(DGLObjectFree+0xb5) [0x7f1316e592c5]<br>
[bt] (7) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f147cfa2dae]<br>
[bt] (8) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x22f) [0x7f147cfa271f]</p>

ptrblck: <p>Could you rerun your code via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and check the stacktrace for an error message or post it here, please?</p>

Haron_Wan: <p>I tried that CUDA_LAUNCH_BLOCKING = 1 code, and it not working. Here is the whole stack trace. Thanks a lot for your help. Let me know if you need more details.</p>
<p>users@amax:~$ python3 -u /home/disk1/users/MyGat/MyGat0417/gat.py --gpu=0<br>
Using backend: pytorch<br>
/home/disk1/users/MyGat/MyGat0417/gat.py --gpu=0<br>
Loading dataset<br>
done<br>
Total edges before adding self-loop 16148<br>
Total edges after adding self-loop 18301<br>
running 1:   0%|                                        | 0/200 [00:00&lt;?, ?it/s]/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [4,0,0], thread: [52,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
running 1:   0%|                                        | 0/200 [00:00&lt;?, ?it/s]<br>
Traceback (most recent call last):<br>
File “/home/disk1/users/MyGat/MyGat0417/gat.py”, line 491, in <br>
main()<br>
File “/home/disk1/users/MyGat/MyGat0417/gat.py”, line 457, in main<br>
log_file, tensorboard_writer, pred_mx)<br>
File “/home/disk1/users/MyGat/MyGat0417/gat.py”, line 275, in run<br>
acc = compute_acc(pred[train_idx], labels[train_idx], evaluator, pred_mx=pred_mx, pred_idx=pred_idx, train_idx=train_idx, importance=True)<br>
File “/home/disk1/users/MyGat/MyGat0417/gat.py”, line 143, in compute_acc<br>
result = evaluator.eval({“y_pred”: y_pred, “y_true”: y_true})<br>
File “/home/disk1/users/MyGat/MyGat0417/utils.py”, line 144, in eval<br>
y_true, y_pred = self._parse_and_check_input(input_dict)<br>
File “/home/disk1/users/MyGat/MyGat0417/utils.py”, line 114, in _parse_and_check_input<br>
y_true = y_true.detach().cpu().numpy()<br>
RuntimeError: CUDA error: device-side assert triggered<br>
terminate called after throwing an instance of ‘dmlc::Error’<br>
what():  [15:06:27] /opt/dgl/src/runtime/cuda/cuda_device_api.cc:103: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: CUDA: device-side assert triggered<br>
Stack trace:<br>
[bt] (0) /home/disk1/users/.local/lib/python3.6/site-packages/dgl/libdgl.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x4f) [0x7f021641d2cf]<br>
[bt] (1) /home/disk1/users/.local/lib/python3.6/site-packages/dgl/libdgl.so(dgl::runtime::CUDADeviceAPI::FreeDataSpace(DLContext, void*)+0x15c) [0x7f0216c6a4ec]<br>
[bt] (2) /home/disk1/users/.local/lib/python3.6/site-packages/dgl/libdgl.so(dgl::runtime::NDArray::Internal::DefaultDeleter(dgl::runtime::NDArray::Container*)+0x1ad) [0x7f0216b284fd]<br>
[bt] (3) /home/disk1/users/.local/lib/python3.6/site-packages/dgl/libdgl.so(dgl::UnitGraph::COO::~COO()+0x127) [0x7f0216c44a37]<br>
[bt] (4) /home/disk1/users/.local/lib/python3.6/site-packages/dgl/libdgl.so(dgl::UnitGraph::~UnitGraph()+0x1ba) [0x7f0216c447ca]<br>
[bt] (5) /home/disk1/users/.local/lib/python3.6/site-packages/dgl/libdgl.so(dgl::HeteroGraph::~HeteroGraph()+0x119) [0x7f0216b414e9]<br>
[bt] (6) /home/disk1/users/.local/lib/python3.6/site-packages/dgl/libdgl.so(DGLObjectFree+0xb5) [0x7f0216aff2c5]<br>
[bt] (7) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f037cbc3dae]<br>
[bt] (8) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x22f) [0x7f037cbc371f]</p>
<p>Aborted (core dumped)</p>

ptrblck: <p>Thanks for the update!<br>
Running the script via <code>CUDA_LAUNCH_BLOCKING=1</code> won’t “fix” the issue, but should give you a proper error message with the failing operation, which is also the case for you:</p>
<pre><code class="lang-python">/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [4,0,0], thread: [52,0,0] Assertion index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds" failed.
</code></pre>
<p>Based on this an indexing operation fails, so you would need to check, where this indexing is used in your code and what values are used for the index, as they are out of bounds.</p>

Harvell36: <p>Thank you for updating us with the outcome!</p>