RuntimeError: device >= 0 && device < num_gpus INTERNAL ASSERT FAILED
Labels: module: cuda, triaged

### 🐛 Describe the bug

It throws `RuntimeError: device >= 0 && device < num_gpus INTERNAL ASSERT FAILED` when I call `torch.cuda.current_device()`.

```python
import torch

torch.cuda.current_device()
```

stacktrace:
```
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/torch/cuda/__init__.py", line 260, in _lazy_init
    queued_call()
  File "/opt/conda/lib/python3.9/site-packages/torch/cuda/__init__.py", line 145, in _check_capability
    capability = get_device_capability(d)
  File "/opt/conda/lib/python3.9/site-packages/torch/cuda/__init__.py", line 381, in get_device_capability
    prop = get_device_properties(device)
  File "/opt/conda/lib/python3.9/site-packages/torch/cuda/__init__.py", line 399, in get_device_properties
    return _get_device_properties(device)  # type: ignore[name-defined]
RuntimeError: device >= 0 && device < num_gpus INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1682343997789/work/aten/src/ATen/cuda/CUDAContext.cpp":50, please report a bug to PyTorch.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.9/site-packages/torch/cuda/__init__.py", line 674, in current_device
    _lazy_init()
  File "/opt/conda/lib/python3.9/site-packages/torch/cuda/__init__.py", line 264, in _lazy_init
    raise DeferredCudaCallError(msg) from e
torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: device >= 0 && device < num_gpus INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1682343997789/work/aten/src/ATen/cuda/CUDAContext.cpp":50, please report a bug to PyTorch.

CUDA call was originally invoked at:

['  File "<stdin>", line 1, in <module>\n', '  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load\n', '  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked\n', '  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked\n', '  File "<frozen importlib._bootstrap_external>", line 850,
 in exec_module\n', '  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed\n', '  File "/opt/conda/lib/python3.9/site-packages/torch/__init__.py", line 1146, in <module>\n    _C._initExtension(manager_path())\n', '  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load\n', '  File "<frozen imp
ortlib._bootstrap>", line 986, in _find_and_load_unlocked\n', '  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked\n', '  File "<frozen importlib._bootstrap_external>", line 850, in exec_module\n', '  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed\n', '  File "/opt/conda/lib/python3.9/
site-packages/torch/cuda/__init__.py", line 197, in <module>\n    _lazy_call(_check_capability)\n', '  File "/opt/conda/lib/python3.9/site-packages/torch/cuda/__init__.py", line 195, in _lazy_call\n    _queued_calls.append((callable, traceback.format_stack()))\n']
['  File "<stdin>", line 1, in <module>\n', '  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load\n', '  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked\n', '  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked\n', '  File "<frozen importlib._bootstrap_external>", line 8
50, in exec_module\n', '  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed\n', '  File "/opt/conda/lib/python3.9/site-packages/torch/__init__.py", line 1146, in <module>\n    _C._initExtension(manager_path())\n', '  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load\n', '  File "<froz
en importlib._bootstrap>", line 986, in _find_and_load_unlocked\n', '  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked\n', '  File "<frozen importlib._bootstrap_external>", line 850, in exec_module\n', '  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed\n', '  File "/opt/conda/lib/p
ython3.9/site-packages/torch/cuda/__init__.py", line 197, in <module>\n    _lazy_call(_check_capability)\n', '  File "/opt/conda/lib/python3.9/site-packages/torch/cuda/__init__.py", line 195, in _lazy_call\n    _queued_calls.append((callable, traceback.format_stack()))\n']
```



### Versions

Python 3.9.17
Torch version: 2.0.1
GPU: 4 * A100, MIG enabled

Can't run the script because throwing the same exception.

cc @ptrblck