Pytorch 1.8.0 fasterrcnn_resnet50_fpn error

username: Kitsunetic
cooked: <p>My environment is</p>
<ul>
<li>OS: Ubuntu 18.04</li>
<li>GPU: RTX3090</li>
<li>CUDA: CUDA11.2</li>
<li>Pytorch 1.8.0_with_CUDA11.1 stable</li>
</ul>
<pre><code class="lang-auto">from torchvision.models.detection import fasterrcnn_resnet50_fpn

box_model = fasterrcnn_resnet50_fpn(pretrained=True, progress=False).cuda()
xs = torch.rand(2, 3, 1080, 1920, dtype=torch.float32).cuda()
ys = [
  {
    "labels": torch.tensor([1], dtype=torch.int64).cuda(),
    "boxes": torch.tensor([[956.0000, 316.3117, 1134.0000, 838.8275]], 
                          dtype=torch.float32).cuda(),
  },
  {
    "labels": torch.tensor([1], dtype=torch.int64).cuda(),
    "boxes": torch.tensor([[956.0000, 316.3117, 1134.0000, 838.8275]], 
                          dtype=torch.float32).cuda(),
  },
]

box_model(xs, ys)
</code></pre>
<p>It occurs error like this.</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-13-7f582a050256&gt; in &lt;module&gt;
----&gt; 1 box_model(xs, ys)

~/anaconda3/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--&gt; 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

~/anaconda3/envs/torch/lib/python3.7/site-packages/torchvision/models/detection/generalized_rcnn.py in forward(self, images, targets)
     95         if isinstance(features, torch.Tensor):
     96             features = OrderedDict([('0', features)])
---&gt; 97         proposals, proposal_losses = self.rpn(images, features, targets)
     98         detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)
     99         detections = self.transform.postprocess(detections, images.image_sizes, original_image_sizes)

~/anaconda3/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--&gt; 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

~/anaconda3/envs/torch/lib/python3.7/site-packages/torchvision/models/detection/rpn.py in forward(self, images, features, targets)
    363             regression_targets = self.box_coder.encode(matched_gt_boxes, anchors)
    364             loss_objectness, loss_rpn_box_reg = self.compute_loss(
--&gt; 365                 objectness, pred_bbox_deltas, labels, regression_targets)
    366             losses = {
    367                 "loss_objectness": loss_objectness,

~/anaconda3/envs/torch/lib/python3.7/site-packages/torchvision/models/detection/rpn.py in compute_loss(self, objectness, pred_bbox_deltas, labels, regression_targets)
    294         """
    295 
--&gt; 296         sampled_pos_inds, sampled_neg_inds = self.fg_bg_sampler(labels)
    297         sampled_pos_inds = torch.where(torch.cat(sampled_pos_inds, dim=0))[0]
    298         sampled_neg_inds = torch.where(torch.cat(sampled_neg_inds, dim=0))[0]

~/anaconda3/envs/torch/lib/python3.7/site-packages/torchvision/models/detection/_utils.py in __call__(self, matched_idxs)
     55             # randomly select positive and negative examples
     56             perm1 = torch.randperm(positive.numel(), device=positive.device)[:num_pos]
---&gt; 57             perm2 = torch.randperm(negative.numel(), device=negative.device)[:num_neg]
     58 
     59             pos_idx_per_image = positive[perm1]

RuntimeError: radix_sort: failed on 1st step: cudaErrorInvalidDevice: invalid device ordinal
</code></pre>
<p>When I try this code on CPU, it works fine.<br>
After that, I reinstalled <code>Pytorch 1.7.1_with_CUDA11.0 stable</code>, it works fine too.</p>

username: albanD
cooked: <p>cc <a class="mention" href="/u/ptrblck">@ptrblck</a> do you know where this could be coming from?</p>

username: ptrblck
cooked: <p>Haven’t seen this error, but let me reproduce it on a 3090.</p>
<p>EDIT: I was able to reproduce it with the 1.8.0+CUDA11.1 conda binaries and will debug it further.<br>
It’s not failing in a source build, so my first guess is to look into CUB/Thrust.</p>

username: namirinz
cooked: <p>Did you just solve the problem. I faces the same now.</p>
<pre><code class="lang-auto">~/torch_env/lib/python3.8/site-packages/torchvision/models/detection/rpn.py in forward(self, images, features, targets)
    362             labels, matched_gt_boxes = self.assign_targets_to_anchors(anchors, targets)
    363             regression_targets = self.box_coder.encode(matched_gt_boxes, anchors)
--&gt; 364             loss_objectness, loss_rpn_box_reg = self.compute_loss(
    365                 objectness, pred_bbox_deltas, labels, regression_targets)
    366             losses = {

~/torch_env/lib/python3.8/site-packages/torchvision/models/detection/rpn.py in compute_loss(self, objectness, pred_bbox_deltas, labels, regression_targets)
    294         """
    295 
--&gt; 296         sampled_pos_inds, sampled_neg_inds = self.fg_bg_sampler(labels)
    297         sampled_pos_inds = torch.where(torch.cat(sampled_pos_inds, dim=0))[0]
    298         sampled_neg_inds = torch.where(torch.cat(sampled_neg_inds, dim=0))[0]

~/torch_env/lib/python3.8/site-packages/torchvision/models/detection/_utils.py in __call__(self, matched_idxs)
     55             # randomly select positive and negative examples
     56             perm1 = torch.randperm(positive.numel(), device=positive.device)[:num_pos]
---&gt; 57             perm2 = torch.randperm(negative.numel(), device=negative.device)[:num_neg]
     58 
     59             pos_idx_per_image = positive[perm1]

RuntimeError: radix_sort: failed on 1st step: cudaErrorInvalidDevice: invalid device ordinal
</code></pre>
<p>OS: Ubuntu 20.04<br>
GPU: RTX 3080<br>
package: pytorch-1.8.0 with CUDA 11.1</p>

username: Kitsunetic
cooked: <p>I’m just using Pytorch 1.7.1 now.</p>
<p>Thank you</p>

username: Lucas_Tom
cooked: <p>I got the same issue[Kitsunetic], what should I do?</p>

username: ptrblck
cooked: <p>It should be fixed already in the nightly conda binary and pip wheel.<br>
Could you update and check it, please?</p>
<p>CC <a class="mention" href="/u/kitsunetic">@Kitsunetic</a> <a class="mention" href="/u/namirinz">@namirinz</a></p>

username: Space_Boy
cooked: <p>I have the same issue with</p>
<pre><code class="lang-auto">print(torch.__version__) 
1.8.0+cu111
</code></pre>
<p>I have a 3090 RTX and torch 1.8 with cuda 11.1 is the only one compatible with Detectron2. Any idea when it will be fixed <a class="mention" href="/u/ptrblck">@ptrblck</a> ?<br>
Thank you</p>

username: MCvin
cooked: <p>I’m having the same issue with pytorch 1.8.1 and cuda 11.1.<br>
The error is different though:</p>
<pre><code class="lang-auto">~/anaconda3/envs/pytorch-1.8.1/lib/python3.8/site-packages/torchvision/models/detection/_utils.py in __call__(self, matched_idxs)
     43         neg_idx = []
     44         for matched_idxs_per_image in matched_idxs:
---&gt; 45             positive = torch.where(matched_idxs_per_image &gt;= 1)[0]
     46             negative = torch.where(matched_idxs_per_image == 0)[0]
     47 

RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>I works with:</p>
<ul>
<li>pytorch 1.8.1 + cuda 10.2</li>
<li>pytorch 1.7.1 + cuda 11.0</li>
</ul>
<p>Seems like cuda 11.1 is the problem here.<br>
Sorry I couldn’t help more.</p>

username: ptrblck
cooked: <p>The radix_sort is already fixed in the nightly release and PyTorch <code>1.8.1</code>, so you would have to update to one of these versions.</p>
<p><a class="mention" href="/u/mcvin">@MCvin</a> your error seems to be different.<br>
Could you rerun the code via <code>CUDA_LAUNCH_BLOCKING=1 python setup.py args</code> and post the complete stack trace (or create a new topic with your error and this information)?</p>

username: gabridego
cooked: <p>After updating to 1.8.1 I’m not getting the radix_sort error anymore, but I’m getting the same error as <a class="mention" href="/u/mcvin">@MCvin</a>. I’m using the stable version with CUDA 11.1 on Ubuntu 18.04. Everything works when run on CPU.</p>
<pre><code class="lang-auto">  File "~/anaconda3/envs/neo/lib/python3.9/site-packages/pl_bolts/models/detection/faster_rcnn/faster_rcnn_module.py", line 112, in training_step
    loss_dict = self.model(images, targets)
  File "~/anaconda3/envs/neo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "~/anaconda3/envs/neo/lib/python3.9/site-packages/torchvision/models/detection/generalized_rcnn.py", line 97, in forward
    proposals, proposal_losses = self.rpn(images, features, targets)
  File "~/anaconda3/envs/neo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "~/anaconda3/envs/neo/lib/python3.9/site-packages/torchvision/models/detection/rpn.py", line 364, in forward
    loss_objectness, loss_rpn_box_reg = self.compute_loss(
  File "~/anaconda3/envs/neo/lib/python3.9/site-packages/torchvision/models/detection/rpn.py", line 296, in compute_loss
    sampled_pos_inds, sampled_neg_inds = self.fg_bg_sampler(labels)
  File "~/anaconda3/envs/neo/lib/python3.9/site-packages/torchvision/models/detection/_utils.py", line 46, in __call__
    positive = torch.where(matched_idxs_per_image &gt;= 1)[0]
RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>The error follows a long series of messages like:</p>
<pre><code class="lang-auto">/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [0,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
</code></pre>
<p>I guess the problem is with the <code>torch.where</code> operation in <code>torchvision/models/detection/_utils.py</code>. I’ve been able to run my program moving the variable <code>matched_idxs_per_image</code> to CPU in the <code>__call__</code> function of class <code>BalancedPositiveNegativeSampler</code>, like:</p>
<pre><code class="lang-auto">for matched_idxs_per_image in matched_idxs:
    matched_idxs_per_image = matched_idxs_per_image.cpu()
    positive = torch.where(matched_idxs_per_image &gt;= 1)[0]
    negative = torch.where(matched_idxs_per_image == 0)[0]
</code></pre>
<p>from line 44.</p>
<p>Hope that helps.</p>

username: ppwwyyxx
cooked: <p>detectron2 is also seeing an increasing number of CUDA error reports on CUDA&gt;=11.1 + pytorch 1.8.x + RTX30xx: <a href="https://github.com/facebookresearch/detectron2/issues/2837" class="inline-onebox" rel="noopener nofollow ugc">RuntimeError: CUDA error: device-side assert triggered · Issue #2837 · facebookresearch/detectron2 · GitHub</a></p>
<p>Root cause seems to be still randperm: <a href="https://github.com/pytorch/pytorch/issues/55027" class="inline-onebox" rel="noopener nofollow ugc">CUDA error: device-side assert triggered(torch1.8.1+cuda11.1) · Issue #55027 · pytorch/pytorch · GitHub</a></p>

username: chrischoy
cooked: <p>I’m facing the same issue on MinokowskiEngine with pytorch 1.8.X + CUDA 11.X. <a href="https://github.com/NVIDIA/MinkowskiEngine/issues/330" rel="noopener nofollow ugc">Cuda 11.1 - Coordinate manager · Issue #330 · NVIDIA/MinkowskiEngine (github.com)</a></p>

username: oo_o
cooked: <p>Same here, seems to be related to randperm()</p>
<p>Reproducible code:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; device = torch.device("cuda:0")
&gt;&gt;&gt; torch.randperm(29999, device=device)
tensor([13324, 19251, 23333,  ..., 18540, 14502, 26766], device='cuda:0')
&gt;&gt;&gt; torch.randperm(30000, device=device)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  RuntimeError: radix_sort: failed on 1st step: cudaErrorInvalidDevice: invalid device ordinal
</code></pre>
<p>pytorch version: 1.8.0+cu111</p>

username: ptrblck
cooked: <p>Could you update PyTorch to <code>1.8.1</code> or the nightly as described in my previous post, please?</p>

username: Kitsunetic
cooked: <p>Tested on 1.8.1 and nightly with same environment.<br>
At 1.8.1, the error still happened but nightly works fine.<br>
Thank you</p>

username: kehuantiantang
cooked: <p>Same problem happened when install nightly version by pip wheel, 1.9.0.dev20210428+cu111, python 3.8.8</p>
<pre><code class="lang-auto">  File "/opt/conda/envs/fpn/lib/python3.8/site-packages/torchvision/models/detection/rpn.py", line 363, in forward
    loss_objectness, loss_rpn_box_reg = self.compute_loss(
  File "/opt/conda/envs/fpn/lib/python3.8/site-packages/torchvision/models/detection/rpn.py", line 295, in compute_loss
    sampled_pos_inds, sampled_neg_inds = self.fg_bg_sampler(labels)
  File "/opt/conda/envs/fpn/lib/python3.8/site-packages/torchvision/models/detection/_utils.py", line 45, in __call__
    positive = torch.where(matched_idxs_per_image &gt;= 1)[0]
RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<pre><code class="lang-auto">/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:97: operator(): block: [0,0,0], thread: [2,0,0] Assertion `index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
</code></pre>
<p>Same code, and problem solved by  <strong>pytorch 1.7.1 + cuda 11.0</strong>,  I think this may be the problem of cuda 11.1</p>

username: ptrblck
cooked: <p>Did you check the indices, which create the error?<br>
If so, what are the min and max values of the indices and what is the shape of the indexed tensor?</p>

username: pierlj
cooked: <p>I had the same error message as <a class="mention" href="/u/mcvin">@MCvin</a> with pytorch 1.8.1 running faster r-cnn code. I tried a few things to reproduce it. It seems to work fine for small value of <code>n</code>. I tried this while running FRCNN code through vscode debugging. Yet, I was not able to reproduce it outside like <a class="mention" href="/u/oo_o">@oo_o</a> did.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/9/9941135cb514f581631d18b272337673af8b0875.png" alt="pytorch_issue" data-base62-sha1="lRKtz7Tg9PEmk3buTzWJgpiCNyB" width="473" height="105"></p>
<p>It was fixed by updating with nightly build  ‘1.9.0.dev20210502’.</p>
<p>OS: Ubuntu 20.04<br>
GPU: RTX 3090<br>
Pytorch 1.8.1 / Cuda 11.1</p>