PyTorch2.3.0+cu121 fails to see CUDA graphics card

Sam_Tux: <p>Hi, I am using PyTorch with smp for computer vision.<br>
I’ve trained a model using Python3.6 and training worked. Model even did reasonable predictions.</p>
<p>Model stoped training when I’ve moved to<br>
PyTorch:  2.3.0+cu121<br>
Python 3.10.7<br>
CUDA Version: 12.4</p>
<p>Code works on Windows. The only difference is that on Windows I have python3.10.11</p>
<p>I do have two CUDA cards and do training on the second card:<br>
environ[‘CUDA_VISIBLE_DEVICES’] = ‘1’</p>
<p>±----------------------------------------------------------------------------------------+<br>
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |<br>
|-----------------------------------------±-----------------------±---------------------+<br>
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |<br>
|                                         |                        |               MIG M. |<br>
|=========================================+========================+======================|<br>
|   0  NVIDIA GeForce GTX 1080        Off |   00000000:00:06.0 Off |                  N/A |<br>
|  0%   30C    P8              6W /  180W |    1790MiB /   8192MiB |      0%      Default |<br>
|                                         |                        |                  N/A |<br>
±----------------------------------------±-----------------------±---------------------+<br>
|   1  NVIDIA GeForce GTX 1080        Off |   00000000:00:07.0 Off |                  N/A |<br>
|  0%   31C    P0             38W /  180W |       0MiB /   8192MiB |      0%      Default |<br>
|                                         |                        |                  N/A |<br>
±----------------------------------------±-----------------------±---------------------+</p>
<p>However training fails with the message below</p>
<p>torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: device &gt;= 0 &amp;&amp; device &lt; num_gpus INTERNAL ASSERT FAILED at “…/aten/src/ATen/cuda/CUDAContext.cpp”:50, please report a bug to PyTorch. device=, num_gpus=</p>
<p>Full output below:<br>
INFO:albumentations.check_version:A new version of Albumentations is available: 1.4.8 (you have 1.4.7). Upgrade using: pip install --upgrade albumentations<br>
missed model data/ttv_1_5_w_background<br>
Traceback (most recent call last):<br>
File “/home/stepan/.local/lib/python3.10/site-packages/torch/cuda/<strong>init</strong>.py”, line 306, in _lazy_init<br>
queued_call()<br>
File “/home/stepan/.local/lib/python3.10/site-packages/torch/cuda/<strong>init</strong>.py”, line 174, in _check_capability<br>
capability = get_device_capability(d)<br>
File “/home/stepan/.local/lib/python3.10/site-packages/torch/cuda/<strong>init</strong>.py”, line 430, in get_device_capability<br>
prop = get_device_properties(device)<br>
File “/home/stepan/.local/lib/python3.10/site-packages/torch/cuda/<strong>init</strong>.py”, line 448, in get_device_properties<br>
return _get_device_properties(device)  # type: ignore[name-defined]<br>
RuntimeError: device &gt;= 0 &amp;&amp; device &lt; num_gpus INTERNAL ASSERT FAILED at “…/aten/src/ATen/cuda/CUDAContext.cpp”:50, please report a bug to PyTorch. device=, num_gpus=</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>Traceback (most recent call last):<br>
File “/home/stepan/acne_seg_models_dev/train_runner_resnet18_normalized.py”, line 236, in <br>
best_model_filenames[dt] = train(data_root, dt, 0)<br>
File “/home/stepan/acne_seg_models_dev/train_runner_resnet18_normalized.py”, line 188, in train<br>
train_epoch = TrainValidateEpochWithAUX(<br>
File “/home/stepan/acne_seg_models_dev/epoch_with_gm.py”, line 12, in <strong>init</strong><br>
super().<strong>init</strong>(<br>
File “/home/stepan/.local/lib/python3.10/site-packages/segmentation_models_pytorch/utils/train.py”, line 75, in <strong>init</strong><br>
super().<strong>init</strong>(<br>
File “/home/stepan/.local/lib/python3.10/site-packages/segmentation_models_pytorch/utils/train.py”, line 16, in <strong>init</strong><br>
self._to_device()<br>
File “/home/stepan/.local/lib/python3.10/site-packages/segmentation_models_pytorch/utils/train.py”, line 19, in _to_device<br>
self.model.to(self.device)<br>
File “/home/stepan/.local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1173, in to<br>
return self._apply(convert)<br>
File “/home/stepan/.local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 779, in _apply<br>
module._apply(fn)<br>
File “/home/stepan/.local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 779, in _apply<br>
module._apply(fn)<br>
File “/home/stepan/.local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 804, in _apply<br>
param_applied = fn(param)<br>
File “/home/stepan/.local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1159, in convert<br>
return t.to(<br>
File “/home/stepan/.local/lib/python3.10/site-packages/torch/cuda/<strong>init</strong>.py”, line 312, in _lazy_init<br>
raise DeferredCudaCallError(msg) from e<br>
torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: device &gt;= 0 &amp;&amp; device &lt; num_gpus INTERNAL ASSERT FAILED at “…/aten/src/ATen/cuda/CUDAContext.cpp”:50, please report a bug to PyTorch. device=, num_gpus=</p>
<p>CUDA call was originally invoked at:</p>
<p>File “/home/stepan/acne_seg_models_dev/train_runner_resnet18_normalized.py”, line 7, in <br>
from torch.utils.data import DataLoader<br>
File “”, line 1027, in _find_and_load<br>
File “”, line 992, in _find_and_load_unlocked<br>
File “”, line 241, in _call_with_frames_removed<br>
File “”, line 1027, in _find_and_load<br>
File “”, line 992, in _find_and_load_unlocked<br>
File “”, line 241, in _call_with_frames_removed<br>
File “”, line 1027, in _find_and_load<br>
File “”, line 1006, in _find_and_load_unlocked<br>
File “”, line 688, in _load_unlocked<br>
File “”, line 883, in exec_module<br>
File “”, line 241, in _call_with_frames_removed<br>
File “/home/stepan/.local/lib/python3.10/site-packages/torch/<strong>init</strong>.py”, line 1478, in <br>
_C._initExtension(manager_path())<br>
File “”, line 1027, in _find_and_load<br>
File “”, line 1006, in _find_and_load_unlocked<br>
File “”, line 688, in _load_unlocked<br>
File “”, line 883, in exec_module<br>
File “”, line 241, in _call_with_frames_removed<br>
File “/home/stepan/.local/lib/python3.10/site-packages/torch/cuda/<strong>init</strong>.py”, line 238, in <br>
_lazy_call(_check_capability)<br>
File “/home/stepan/.local/lib/python3.10/site-packages/torch/cuda/<strong>init</strong>.py”, line 235, in _lazy_call<br>
_queued_calls.append((callable, traceback.format_stack()))</p>

ptrblck: <p>It seems your system might have issues communicating with your GPU. You could try compiling and running any other CUDA application to verify if you are still seeing the same issue. If so, you might need to reinstall your NVIDIA drivers.</p>

Sam_Tux: <p>I will try. Though it is strange that<br>
(1) Code works on earlier version of Python and Torch (on the same system)<br>
(2) import torch<br>
print(porch.cuda.is_available())  # true</p>