Torch.cuda.get_device_name(0) error response

Frank1: <p>pytorch version: 1.12.0+cu102<br>
cuda version:  release 11.6, V11.6.124</p>
<blockquote>
<blockquote>
<p>torch.cuda.get_device_name(0)<br>
/usr/local/lib/python3.8/dist-packages/torch/cuda/<strong>init</strong>.py:146: UserWarning:<br>
NVIDIA A100-SXM4-80GB with CUDA capability sm_80 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.<br>
If you want to use the NVIDIA A100-SXM4-80GB GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
</blockquote>
</blockquote>
<p>warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))<br>
‘NVIDIA A100-SXM4-80GB’</p>
<blockquote>
<blockquote>
<blockquote>
<p>torch.cuda.get_device_name(1)<br>
‘NVIDIA A100-SXM4-80GB’<br>
torch.cuda.get_device_name(2)<br>
‘NVIDIA A100-SXM4-80GB’<br>
torch.cuda.get_device_name(3)<br>
‘NVIDIA A100-SXM4-80GB’<br>
torch.cuda.get_device_name(4)<br>
‘NVIDIA A100-SXM4-80GB’<br>
torch.cuda.get_device_name(5)<br>
‘NVIDIA A100-SXM4-80GB’<br>
torch.cuda.get_device_name(6)<br>
‘NVIDIA A100-SXM4-80GB’<br>
torch.cuda.get_device_name(7)<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/usr/local/lib/python3.8/dist-packages/torch/cuda/<strong>init</strong>.py”, line 329, in get_device_name<br>
return get_device_properties(device).name<br>
File “/usr/local/lib/python3.8/dist-packages/torch/cuda/<strong>init</strong>.py”, line 362, in get_device_properties<br>
raise AssertionError("Invalid device i</p>
</blockquote>
</blockquote>
</blockquote>

ptrblck: <p>You’ve installed a PyTorch binary with CUDA 10.2, while your A100 Ampere GPU needs CUDA&gt;=11.<br>
Install any of the latest binaries and it’ll work.</p>

Frank1: <p>I update the PyTorch version had the same problem, giid:7 gpu loss</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.cuda.device_count()<br>
7<br>
torch.cuda.get_device_name(0)<br>
‘NVIDIA A100-SXM4-80GB’<br>
torch.cuda.get_device_name(1)<br>
‘NVIDIA A100-SXM4-80GB’<br>
torch.cuda.get_device_name(2)<br>
‘NVIDIA A100-SXM4-80GB’<br>
torch.cuda.get_device_name(3)<br>
‘NVIDIA A100-SXM4-80GB’<br>
torch.cuda.get_device_name(4)<br>
‘NVIDIA A100-SXM4-80GB’<br>
torch.cuda.get_device_name(5)<br>
‘NVIDIA A100-SXM4-80GB’<br>
torch.cuda.get_device_name(6)<br>
‘NVIDIA A100-SXM4-80GB’<br>
torch.cuda.get_device_name(7)<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/opt/conda/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py”, line 329, in get_device_name<br>
return get_device_properties(device).name<br>
File “/opt/conda/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py”, line 362, in get_device_properties<br>
raise AssertionError(“Invalid device id”)<br>
AssertionError: Invalid device id<br>
torch.<strong>version</strong><br>
‘1.12.1+cu113’</p>
</blockquote>
</blockquote>
</blockquote>

ptrblck: <p>Could you describe what <code>device7</code> is according to <code>nvidia-smi</code>?</p>

Frank1: <p>nvidia-smi can display device 7<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: 11.6     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|                               |                      |               MIG M. |<br>
|===============================+======================+======================|<br>
|   0  NVIDIA A100-SXM…  On   | 00000000:10:00.0 Off |                    0 |<br>
| N/A   39C    P0    65W / 400W |      0MiB / 81920MiB |      0%      Default |<br>
|                               |                      |             Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
|   1  NVIDIA A100-SXM…  On   | 00000000:16:00.0 Off |                    0 |<br>
| N/A   36C    P0    66W / 400W |      0MiB / 81920MiB |      0%      Default |<br>
|                               |                      |             Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
|   2  NVIDIA A100-SXM…  On   | 00000000:2F:00.0 Off |                    0 |<br>
| N/A   36C    P0    63W / 400W |      0MiB / 81920MiB |      0%      Default |<br>
|                               |                      |             Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
|   3  NVIDIA A100-SXM…  On   | 00000000:33:00.0 Off |                    0 |<br>
| N/A   38C    P0    69W / 400W |      0MiB / 81920MiB |      0%      Default |<br>
|                               |                      |             Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
|   4  NVIDIA A100-SXM…  On   | 00000000:C5:00.0 Off |                    0 |<br>
| N/A   37C    P0    66W / 400W |      0MiB / 81920MiB |      0%      Default |<br>
|                               |                      |             Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
|   5  NVIDIA A100-SXM…  On   | 00000000:CA:00.0 Off |                    0 |<br>
| N/A   38C    P0    65W / 400W |      0MiB / 81920MiB |      0%      Default |<br>
|                               |                      |             Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
|   6  NVIDIA A100-SXM…  On   | 00000000:E3:00.0 Off |                    0 |<br>
| N/A   36C    P0    67W / 400W |      0MiB / 81920MiB |      0%      Default |<br>
|                               |                      |             Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
|   7  NVIDIA A100-SXM…  On   | 00000000:E7:00.0 Off |                    0 |<br>
| N/A   40C    P0    67W / 400W |      0MiB / 81920MiB |      0%      Default |<br>
|                               |                      |             Disabled |<br>
±------------------------------±---------------------±---------------------+</p>

ptrblck: <p>Are you able to run any code on <code>cuda:7</code>? If not, did you set <code>CUDA_VISIBLE_DEVICES</code> in your environment?</p>