CUDA error: an illegal memory access was encountered - torch.ops.torchvision.roi_align

username: Sueone
cooked: <p>Hey there.<br>
I’m trying to get RoIs by torch.ops.torchvision.roi_align with GPU but getting this <code>CUDA error: an illegal memory access was encountered.</code><br>
I’ve set my device to cuda 1 by <code>torch.cuda.set_device(1)</code>.<br>
Some posts say I should set <code>torch.backends.cudnn.benchmark = False</code>, but mine is already set False.</p>
<p>This is my code snippet.</p>
<pre><code class="lang-auto">aligned = []
for coor, feat_map in zip(coordinates, feature_maps):
     aligned.append(roi_align(feat_map, coor, output_size=7))
</code></pre>
<p>and I get the error from the <code>coordinates[7]</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c24b7268994a73717dec7c12fb3b0c6971cad6d8.png" data-download-href="https://discuss.pytorch.org/uploads/default/c24b7268994a73717dec7c12fb3b0c6971cad6d8" title="스크린샷, 2021-09-08 20-01-06"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c24b7268994a73717dec7c12fb3b0c6971cad6d8_2_690x300.png" alt="스크린샷, 2021-09-08 20-01-06" data-base62-sha1="rIOgXqezPAz5iJSHu7T5XvdXv04" width="690" height="300" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c24b7268994a73717dec7c12fb3b0c6971cad6d8_2_690x300.png, https://discuss.pytorch.org/uploads/default/original/3X/c/2/c24b7268994a73717dec7c12fb3b0c6971cad6d8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/2/c24b7268994a73717dec7c12fb3b0c6971cad6d8.png 2x" data-dominant-color="464C53"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">스크린샷, 2021-09-08 20-01-06</span><span class="informations">751×327 64.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
My feature maps are {list:10} and the shape varies. (e.g. [1, 512, 12, 16], [1, 512, 11, 16], [1, 512, 16, 14]…)<br>
This is the original roi_align codes and the error occurs at the <code>return ~</code> line.</p>
<pre><code class="lang-auto">def roi_align(
    input: Tensor,
    boxes: Tensor,
    output_size: BroadcastingList2[int],
    spatial_scale: float = 1.0,
    sampling_ratio: int = -1,
    aligned: bool = False,
) -&gt; Tensor:
         _assert_has_ops()
         check_roi_boxes_shape(boxes)
         rois = boxes
         output_size = _pair(output_size)
         if not isinstance(rois, torch.Tensor):
             rois = convert_boxes_to_roi_format(rois)
         return torch.ops.torchvision.roi_align(input, rois, spatial_scale,
                                           output_size[0], output_size[1],
                                           sampling_ratio, aligned)
</code></pre>
<p>I’ve runned the code with the<code>CUDA_LAUNCH_BLOCKING=1</code>also and got this.</p>
<pre><code class="lang-auto">❯ CUDA_LAUNCH_BLOCKING=1 python usage.py
Use load_from_local loader
The model and loaded state dict do not match exactly

unexpected key in source state_dict: roi_head.bbox_head.fc_cls.weight, roi_head.bbox_head.fc_reg.weight, roi_head.bbox_head.fc_reg.bias, roi_head.bbox_head.shared_fcs.0.weight, roi_head.bbox_head.shared_fcs.0.bias, roi_head.bbox_head.shared_fcs.1.weight, roi_head.bbox_head.shared_fcs.1.bias

/home/sueyeon/Projects/mmdetection/mmdet/datasets/utils.py:68: UserWarning: "ImageToTensor" pipeline is replaced by "DefaultFormatBundle" for batch inference. It is recommended to manually replace it in the test data pipeline in your config file.
  'data pipeline in your config file.', UserWarning)
Traceback (most recent call last):
  File "usage.py", line 284, in &lt;module&gt;
    cropped_gt_support_rois = crop_rois(revised_support_gt_coordinates, unchanged_img_supports)
  File "usage.py", line 174, in crop_rois
    aligned.append(roi_align(feat_map, coor, output_size=7))
  File "/home/sueyeon/anaconda3/envs/ctx/lib/python3.7/site-packages/torchvision/ops/roi_align.py", line 55, in roi_align
    sampling_ratio, aligned)
RuntimeError: CUDA error: an illegal memory access was encountered
</code></pre>
<p>To check if it is the GPU specific problem, I’ve unloaded all the tensors from GPU and runned the codes on the CPU and it worked! However it returned nan values and inf values which is not ideal… I need to train the model with GPUs anyway so I need to solve this problem <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=12" title=":cry:" class="emoji" alt=":cry:" loading="lazy" width="20" height="20"> Hope somebody please can help me out … <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>Are you seeing the same issue if you are using the default device (GPU0)?<br>
If not, it would sound like a missing device guard, which seems to be used <a href="https://github.com/pytorch/vision/blob/7947fc8fb38b1d3a2aca03f22a2e6a3caa63f2a0/torchvision/csrc/ops/cuda/roi_align_kernel.cu#L336">here</a> however.</p>