Get "RuntimeError: CUDA error: device-side assert triggered"

yanglongfei908: <p>Snippet from my code :</p>
<pre><code class="lang-auto">max = torch.tensor([3])
if USE_CUDA: max = max.cuda()
max_embedding = self.max_embedding(max) # dim of max_embedding: 1*5

item_dict = {}
for item in item_list:
    item = torch.tensor(item)
    if USE_CUDA: item = item.cuda()
    item_embedding = self.item_embedding(item) # dim of item_embedding: 1*20

embedded = torch.cat((max_embedding, item_embedding), 1)
</code></pre>
<p>But I get error of <code>RuntimeError: CUDA error: device-side assert triggered"</code>.<br>
The output by adding <code>CUDA_LAUNCH_BLOCKING=1</code>:</p>
<pre><code class="lang-auto">/pytorch/aten/src/THC/THCTensorIndex.cu:308: void indexSelectSmallIndex(TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [0,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [1,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [2,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [3,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [4,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.
Traceback (most recent call last):
  File "mytest.py", line 33, in &lt;module&gt;
     if USE_CUDA: item = item.cuda()
RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>How to fix it?</p>

ptrblck: <p>Some indexing operation fails and I assume it’s related to your <code>nn.Embedding</code> layer.<br>
Is your code running fine on the CPU? This might generally give you a better error message.</p>

yanglongfei908: <p>Yeah, I find it. The size of vocab in <code>max_embedding</code> layer should be <code>maximum index + 1</code> but mine is <code>maximum index</code>.<br>
But why the Traceback indicate the error in other line of <code>if USE_CUDA: item = item.cuda()</code> over show the line of <code>max_embedding = self.max_embedding(max)</code>?</p>

ptrblck: <p>Since CUDA operations are asynchronous, the stack trace might indeed point to a wrong line of code.<br>
This can usually be fixed by running the script via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code>, which will synchronize all CUDA operations or on the CPU.</p>

aswamy: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I am stuck with the same error for a different code repository.  I saw your suggestion to add CUDA_LAUNCH_BLOCKING=1 in the script and then run the script. I tried and the error still persists. Complete information about the error and the trace is available in this <a href="https://github.com/krrish94/nerf-pytorch/issues/22" rel="noopener nofollow ugc">link</a></p>
<p>Do you have hints to solve this problem? Please let me know if you need any more information.</p>
<p>It runs without any error on CPU</p>
<p>Thank you!</p>

ptrblck: <p>The stack trace from the linked issue points to this failing <code>scatter</code> call:</p>
<pre><code class="lang-python">cdf_g = torch.gather(cdf.unsqueeze(1).expand(matched_shape), 2, inds_g))
==============================================================
/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [47,0,0], thread: [47,0,0] Ass ertion idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds" failed.
</code></pre>
<p>so you would have to check the tensor shapes and index values.<br>
Running the code on the CPU could also yield a better error message.</p>

aswamy: <p>Code runs successfully on CPU!<br>
Yes, I have checked the index values, they are very <a href="https://github.com/krrish94/nerf-pytorch/issues/22#issuecomment-746044962" rel="noopener nofollow ugc">extreme</a></p>
<p>I tried to print the tensors and found that values of tensor inds_g is too large and too small(which casuses out of bounds error)<br>
inds_g.min() = tensor(-4993021444723710459)<br>
inds_g.max() = tensor(4575432887736600530)<br>
inds_g = tensor([[[ 4255818524050935954, 62],<br>
[ 4256250760978027750, 62],<br>
[ 4237722774569238629, 62],<br>
…,]]</p>
<p>usage of this tensor:<br>
file “run_nerf_helpers.py”<br>
cdf_g = torch.gather(cdf.unsqueeze(1).expand(matched_shape), 2, inds_g)</p>

aswamy: <p>There is not dimension problem, its due to extreme values of inds_g variable that is causing out of  bound error in torch.gather(). I validated this by setting inds_g to zero, then code runs on gpu as well.<br>
But I do not why on gpu values are computed to extreme.</p>

ptrblck: <p>For clarity and potentially other users encountering this error: Based on the linked issue it seems to be caused by <code>torchsearchsorted.searchsorted()</code> and the PyTorch implementation solves the issue.</p>