Overflow and get a very large number when using torch.topk

zhoulukuan: <p>When I use torch.topk to get the index, I find that the program will stop quickly after several training iters because of <code>Pyorch-CUDA error: device-side assert triggered, THCTensorScatterGather, Assertion indexValue failed</code>.<br>
I printed the index and found sometimes torch.topk will get a very large number such as tensor(9223372034707292159, device=‘cuda:0’) and as a results, the program cannot get the corresponeding data by this large index. How to solve this overlflow problem?</p>

ptrblck: <p>Could you post a code snippet to reproduce this error, please?<br>
I assume you are using <code>float32</code> tensors?<br>
If you cannot post the code, could you post the shape of the input tensor and its range?</p>

zhoulukuan: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  The code is changed from <a href="https://github.com/open-mmlab/mmdetection/blob/master/mmdet/models/anchor_heads/free_anchor_retina_head.py" rel="nofollow noopener">https://github.com/open-mmlab/mmdetection/blob/master/mmdet/models/anchor_heads/free_anchor_retina_head.py</a>.</p>
<pre><code class="lang-auto">match_matrix = bboxoverlaps(gt_bboxes, bbox_preds)
# match_matrix: torch.float32, shapes: [num_gts, num_anchors]
matched_iou, matched = torch.topk(match_matrix, 50, dim=1, sorted=False)

# cls_prob:  torch.float32, shapes: [num_anchors, 80]
temp = cls_prob[matched]
</code></pre>
<p>I have some difficulty reproducing this error, because this error happened very accidentally. Sometimes, it will appear immediately after training, sometimes, a few epochs it will appear, and sometimes, I can complete the training process. I am very distressed.</p>

ptrblck: <p>Just to clarify, <code>match_matrix</code> contains these invalid values (<code>9223372034707292159</code>) sometimes and lets <code>topk</code> crash.<br>
If that’s the case, it seems that <code>bboxoverlaps</code> is creating wrong results sometimes and I would suggest to check the indexing in the method.</p>

zhoulukuan: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  It means I need to check the <code>bbooverlaps</code> results? How to understand wrong results, such as NAN? Is there any other situation？<br>
Besides, when I set a breakpoint, I found that once the above error occurs, all variables will become <code>Unable to get repr for &lt;class 'torch.Tensor'&gt;</code>, cannot be viewed, what should I do?</p>

ptrblck: <aside class="quote no-group" data-username="zhoulukuan" data-post="5" data-topic="78253">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhoulukuan/48/12368_2.png" class="avatar"> zhoulukuan:</div>
<blockquote>
<p>It means I need to check the <code>bbooverlaps</code> results?</p>
</blockquote>
</aside>
<p>Yes. I’m not sure what tensors are passed to this method, but the calculation might be wrong for “unexpected” tensors or you might run into an overflow.</p>
<aside class="quote no-group" data-username="zhoulukuan" data-post="5" data-topic="78253">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhoulukuan/48/12368_2.png" class="avatar"> zhoulukuan:</div>
<blockquote>
<p>How to understand wrong results, such as NAN? Is there any other situation？</p>
</blockquote>
</aside>
<p>NaN values can be created in various ways. Based on the posted tensor value, it doesn’t seem that you are running into NaNs but invalid v alues.</p>
<aside class="quote no-group" data-username="zhoulukuan" data-post="5" data-topic="78253">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhoulukuan/48/12368_2.png" class="avatar"> zhoulukuan:</div>
<blockquote>
<p>Besides, when I set a breakpoint, I found that once the above error occurs, all variables will become <code>Unable to get repr for &lt;class 'torch.Tensor'&gt;</code> , cannot be viewed, what should I do?</p>
</blockquote>
</aside>
<p>I’m not sure which debugger you are using, but you could add <code>assert</code> statements (or conditions) and print the desired values.</p>

Boltzmachine: <p>I also encounter this issue. The tensor does not contain NaNs.<br>
The solution is to run the same code twice… and I do not know why it works</p>
<pre><code class="lang-auto">idx = tensor.topk(...)
idx = tensor.topk(...)
</code></pre>
<p>And if I run it on CPU, it works</p>

beny-maleki: <p>I have also run into the same issue! When I run the same line twice, the second result has valid values that are in the expected index range. The first result consists of strangely large values. Is this some sort of bug?</p>

Xiuchen519: <p>Also encountered this problem. I reduce the batch size and it works. It may be caused by OOM, but there is no OOM error reported.</p>