RuntimeError: handle_0 INTERNAL ASSERT FAILED at "../c10/cuda/driver_api.cpp":15, please report a bug to PyTorch

username: Iruma_Megomi
cooked: <p>Bug:<br>
Traceback (most recent call last):<br>
File “/media/iruma/Data/AIR/radar/OpenPCDet/tools/train.py”, line 230, in <br>
main()<br>
File “/media/iruma/Data/AIR/radar/OpenPCDet/tools/train.py”, line 175, in main<br>
train_model(<br>
File “/media/iruma/Data/AIR/radar/OpenPCDet/tools/train_utils/train_utils.py”, line 180, in train_model<br>
accumulated_iter = train_one_epoch(<br>
File “/media/iruma/Data/AIR/radar/OpenPCDet/tools/train_utils/train_utils.py”, line 56, in train_one_epoch<br>
loss, tb_dict, disp_dict = model_func(model, batch)<br>
File “/media/iruma/Data/AIR/radar/OpenPCDet/tools/…/pcdet/models/<strong>init</strong>.py”, line 44, in model_func<br>
ret_dict, tb_dict, disp_dict = model(batch_dict)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1527, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/media/iruma/Data/AIR/radar/OpenPCDet/tools/…/pcdet/models/detectors/pv_rcnn.py”, line 11, in forward<br>
batch_dict = cur_module(batch_dict)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1527, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/media/iruma/Data/AIR/radar/OpenPCDet/tools/…/pcdet/models/roi_heads/pvrcnn_head.py”, line 151, in forward<br>
pooled_features = self.roi_grid_pool(batch_dict)  # (BxN, 6x6x6, C)<br>
File “/media/iruma/Data/AIR/radar/OpenPCDet/tools/…/pcdet/models/roi_heads/pvrcnn_head.py”, line 97, in roi_grid_pool<br>
pooled_points, pooled_features = self.roi_grid_pool_layer(<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1527, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/media/iruma/Data/AIR/radar/OpenPCDet/tools/…/pcdet/ops/pointnet2/pointnet2_stack/pointnet2_modules.py”, line 95, in forward<br>
new_features = self.mlps<a>k</a>  # (1, C, M1 + M2 …, nsample)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1527, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py”, line 215, in forward<br>
input = module(input)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1527, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/activation.py”, line 101, in forward<br>
return F.relu(input, inplace=self.inplace)<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py”, line 1471, in relu<br>
result = torch.relu(input)<br>
RuntimeError: handle_0 INTERNAL ASSERT FAILED at “…/c10/cuda/driver_api.cpp”:15, please report a bug to PyTorch.<br>
My environment:<br>
Pytorch version: 2.0.1<br>
CUDA version:12.1<br>
OS: Ubuntu 22.04<br>
Python version:python 3.10</p>

username: ptrblck
cooked: <p>Could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a>, please?</p>

username: Iruma_Megomi
cooked: <aside class="onebox githubissue" data-onebox-src="https://github.com/open-mmlab/OpenPCDet/issues/1489">
  <header class="source">

      <a href="https://github.com/open-mmlab/OpenPCDet/issues/1489" target="_blank" rel="noopener nofollow ugc">github.com/open-mmlab/OpenPCDet</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/open-mmlab/OpenPCDet/issues/1489" target="_blank" rel="noopener nofollow ugc">RuntimeError: handle_0 INTERNAL ASSERT FAILED at "../c10/cuda/driver_api.cpp":15, please report a bug to PyTorch.</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2023-10-16" data-time="14:11:56" data-timezone="UTC">02:11PM - 16 Oct 23 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/IrumaMegumi" target="_blank" rel="noopener nofollow ugc">
          <img alt="IrumaMegumi" src="https://avatars.githubusercontent.com/u/50224448?v=4" class="onebox-avatar-inline" width="20" height="20">
          IrumaMegumi
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Traceback (most recent call last):
  File "/media/iruma/Data/AIR/radar/OpenPCDe<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">t/tools/train.py", line 230, in &lt;module&gt;
    main()
  File "/media/iruma/Data/AIR/radar/OpenPCDet/tools/train.py", line 175, in main
    train_model(
  File "/media/iruma/Data/AIR/radar/OpenPCDet/tools/train_utils/train_utils.py", line 180, in train_model
    accumulated_iter = train_one_epoch(
  File "/media/iruma/Data/AIR/radar/OpenPCDet/tools/train_utils/train_utils.py", line 56, in train_one_epoch
    loss, tb_dict, disp_dict = model_func(model, batch)
  File "/media/iruma/Data/AIR/radar/OpenPCDet/tools/../pcdet/models/__init__.py", line 44, in model_func
    ret_dict, tb_dict, disp_dict = model(batch_dict)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/iruma/Data/AIR/radar/OpenPCDet/tools/../pcdet/models/detectors/pv_rcnn.py", line 11, in forward
    batch_dict = cur_module(batch_dict)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/iruma/Data/AIR/radar/OpenPCDet/tools/../pcdet/models/roi_heads/pvrcnn_head.py", line 151, in forward
    pooled_features = self.roi_grid_pool(batch_dict)  # (BxN, 6x6x6, C)
  File "/media/iruma/Data/AIR/radar/OpenPCDet/tools/../pcdet/models/roi_heads/pvrcnn_head.py", line 97, in roi_grid_pool
    pooled_points, pooled_features = self.roi_grid_pool_layer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/iruma/Data/AIR/radar/OpenPCDet/tools/../pcdet/ops/pointnet2/pointnet2_stack/pointnet2_modules.py", line 95, in forward
    new_features = self.mlps[k](new_features)  # (1, C, M1 + M2 ..., nsample)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py", line 215, in forward
    input = module(input)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/activation.py", line 101, in forward
    return F.relu(input, inplace=self.inplace)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 1471, in relu
    result = torch.relu(input)
RuntimeError: handle_0 INTERNAL ASSERT FAILED at "../c10/cuda/driver_api.cpp":15, please report a bug to PyTorch.

When I use the train.py code, I found the error. I don't know how to solve it. Please help me. Thanks.</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
My issue is here.</p>