Can't initialize NVML: Ambiguous response

tovtyvold: <p>Hi all,</p>
<p>We have been running a lot of VM’s containerized using docker with the docker base image:<br>
<code>pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime</code><br>
and the script:</p>
<pre><code class="lang-auto">import subprocess
import torch


if __name__ == '__main__':
    print('Testing NVIDIA SMI')
    try:
        # Run the nvidia-smi command
        result = subprocess.run(['nvidia-smi'], capture_output=True, text=True, check=True)
        # Print the output
        print("Result", result)
        print("Result stdout: ", result.stdout)
    except subprocess.CalledProcessError as e:
        print(f"subprocess.CalledProcessError occurred: {e}")
    print('Testing CUDA')
    try:
        print("CUDA CHECK ---------------------")
        print('CUDA available running "torch.cuda.is_available()":', torch.cuda.is_available())
        print('CUDA device count:"torch.cuda.device_count()"', torch.cuda.device_count())
        print('CUDA device name:', torch.cuda.get_device_name(0))
        if torch.cuda.is_available():
            for idx in range(torch.cuda.device_count()):
                name = torch.cuda.get_device_name(idx)
                cap = torch.cuda.get_device_capability(idx)
                mem = torch.cuda.get_device_properties(idx).total_memory/1e9
            print(f"{name}  (sm{cap[0]}{cap[1]})  {mem:.1f} GB")
    except Exception as e:
        print(f"Error occurred: {e}")

</code></pre>
<p>and the scripts ran are <em>mostly</em> succesful, but sometimes fail. It fails about 10% of the time and throws the following:</p>
<pre><code class="lang-auto">/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py:789: UserWarning: Can't initialize NVML
  warnings.warn("Can't initialize NVML")
</code></pre>
<p>as you also can see, this is thrown before the script is ran, which suggest something is off with docker or the host machine. Note that this <strong>usually</strong> runs fine and happens randomly.</p>
<p>Exactly what the error is, is difficult to say. Looking at the codebase <a href="https://github.com/pytorch/pytorch/blob/d76323d41742cbc05ec6857319b267d2c7ea8fd9/torch/cuda/__init__.py#L827" rel="noopener nofollow ugc">here</a>(there is another occurence as well) and the docs for what nvmlInit <a href="https://developer.download.nvidia.com/compute/DevZone/NVML/doxygen/group__group1.html" rel="noopener nofollow ugc">function returns</a>, I do not know why it fails. Whether I should post this here or make a Github issue, I do not know.</p>
<p>I am asking for some guidance on this though. We want stability on our VM’s, but this sudden occurrence of failing to initialize causes us trouble. Any help would be greatly appreciated!</p>
<p>Cheers,<br>
Tov</p>

tovtyvold: <p>I have ran some more tests which seems to indicate that something strange is going on with pytorch. This is how the output look after running the script on a node(cleaned up a bit for readability):</p>
<pre><code class="lang-auto">Testing NVIDIA SMI
Result stdout:  Thu Jul 17 11:38:43 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20             Driver Version: 570.133.20     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 NVL                On  |   00000001:00:00.0 Off |                    0 |
| N/A   34C    P0             62W /  400W |       0MiB /  95830MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 NVL                On  |   00000002:00:00.0 Off |                    0 |
| N/A   33C    P0             62W /  400W |       0MiB /  95830MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Testing CUDA
CUDA CHECK ---------------------
CUDA available running "torch.cuda.is_available()": True
CUDA device count:"torch.cuda.device_count()" 2
CUDA device name: NVIDIA H100 NVL
NVIDIA H100 NVL  (sm90)  100.0 GB
</code></pre>
<p>and this is the output on an identical VM(but not the same):</p>
<pre><code class="lang-auto">Testing NVIDIA SMI
Result stdout:  Thu Jul 17 11:39:15 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20             Driver Version: 570.133.20     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 NVL                On  |   00000001:00:00.0 Off |                    0 |
| N/A   35C    P0             63W /  400W |       0MiB /  95830MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 NVL                On  |   00000002:00:00.0 Off |                    0 |
| N/A   34C    P0             62W /  400W |       0MiB /  95830MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Testing CUDA
CUDA CHECK ---------------------
CUDA available running "torch.cuda.is_available()": False
CUDA device count:"torch.cuda.device_count()" 0
Error occurred: No CUDA GPUs are available
</code></pre>
<p>As far as I know, this does not make any sense. Do I have to explicitly set something to make sure that pytorch knows about the GPUs?</p>

ptrblck: <p>No, PyTorch will use the CUDA runtime to detect GPUs and is most likely not causing the issues you are seeing. You could run any standalone CUDA application (e.g. from the CUDA samples or any custom code) on the failing node to further isolate why GPUs cannot be detected anymore.</p>

tovtyvold: <p>Thanks for replying. Since the command nvidia-smi is able to detect the GPU’s, does that imply that CUDA <em>should</em> also be able to detect the GPU’s? That is, something is going wrong with the CUDA toolkit/installation or in any way, it is a CUDA issue?</p>

ptrblck: <p>No, it doesn’t since the driver could still run into issues which would be raised during the runtime.</p>

tovtyvold: <p>Ok, so the nvidia drivers cannot be that easily disregarded as the potential issue.<br>
Could you by chance point me to some sample code that could be used to debug the issue?</p>
<p>I still think that <a href="https://github.com/pytorch/pytorch/blob/d76323d41742cbc05ec6857319b267d2c7ea8fd9/torch/cuda/__init__.py#L827" rel="noopener nofollow ugc">the parsing of the potential error</a> could be improved for the consumer(me), as I do not have a really good idea on how to approach that. The reason I am saying that is that I have a hunch that it might be a re-initializing error, but I have no way of knowing from the response without doing a lot of work, even though it could be presented.</p>

ptrblck: <p>Any example from <a href="https://github.com/NVIDIA/cuda-samples"><code>cuda-samples</code></a> should work as another test assuming you are using a locally installed CUDA toolkit and are able to build these tests.</p>
<p>Yes, thanks for the feedback as we should check if the error message can be improved.</p>

tovtyvold: <p>Ok, I will perform some testing. Thanks for the help!</p>

tovtyvold: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>I have a question regarding the implementation in torch; why does it never call nvmlShutdown()? Is it up to the user to properly shut it down after it is done communicating with the GPU’s?</p>
<p>Edit: I think I understand that short lived tasks does not need to call nvmlShutdown, as a fresh instance of the host will probably be present. In our case though, we most likely will not have a host which has most of its services reset. Since whenever pytorch runs, it does not really release the nvml resources it has requested, could that be of importance to understanding our error?</p>

ptrblck: <aside class="quote no-group" data-username="tovtyvold" data-post="9" data-topic="221604">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/bbce88/48.png" class="avatar"> tovtyvold:</div>
<blockquote>
<p>Is it up to the user to properly shut it down after it is done communicating with the GPU’s?</p>
</blockquote>
</aside>
<p>Once the Python process terminates, the CUDARuntime will shut down all resources including NVML as well as any CUDA Math lib handles. PyTorch itself does not provide interfaces to shutdown the CUDA runtime or reset the CUDA Context etc. manually.</p>
<aside class="quote no-group" data-username="tovtyvold" data-post="9" data-topic="221604">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/bbce88/48.png" class="avatar"> tovtyvold:</div>
<blockquote>
<p>I think I understand that short lived tasks does not need to call nvmlShutdown, as a fresh instance of the host will probably be present. In our case though, we most likely will not have a host which has most of its services reset. Since whenever pytorch runs, it does not really release the nvml resources it has requested, could that be of importance to understanding our error?</p>
</blockquote>
</aside>
<p>I don’t understand this description and and e.g. what “fresh instance of the host” means. No, I don’t think any of our NVML usage is related to the issues you are seeing as I assume also plain CUDA samples will also break, which you did not confirm yet.</p>

tovtyvold: <p>I did not know that the CUDARuntime will respond in that manner, thanks for clarifying that. I know that Pytorch does not provide interfaces for that, which led to my questioning.</p>
<p>I was underway to try out the CUDA samples, but it has not been easy to automate. The reason I want to automate it is because CUDARuntime only fails 1/10 times and our nodes are quickly deallocated to save resources. But I will get around to test it, it seems like a really good test to perform.</p>

tovtyvold: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Hi again,</p>
<p>I have now ran the samples on the machine. Before testing I verified whether the node could run the usual torch commands it fails on and in this case the machine can run ```torch.cuda.is_available()```  succesfully.</p>
<p>Regardless, the output samples give the following output during building:</p>
<p>`nvcc fatal   : Unsupported gpu architecture ‘compute_110’`</p>
<p>multiple times. The build also fails later on due to some invalid conversion being performed. Which does not look promising. This does not belong in this forum, but while we are at if you were to have more experience with this:</p>
<p>I have ran nvcc -V:<br>
Cuda compilation tools, release 12.8, V12.8.93<br>
Build cuda_12.8.r12.8/compiler.35583870_0</p>
<p>and in /usr/local/cuda-12.8/version.json:</p>
<p>“cuda” : {<br>
“name” : “CUDA SDK”,<br>
“version” : “12.8.1”<br>
},</p>
<p>and</p>
<p>“cuda_nvcc” : {<br>
“name” : “CUDA NVCC”,<br>
“version” : “12.8.93”<br>
},</p>
<p>also, during one of the test in the Samples(I ran the 1_utilities isolated), I got the following output:</p>
<p>Device 3: “Tesla T4”<br>
CUDA Driver Version / Runtime Version          12.8 / 12.8<br>
CUDA Capability Major/Minor version number:    7.5</p>
<p>Now, searching the web, CUDA 12.8 should support the Turing machines. But I do not know why it is looking for compute 110, and if it is the CUDA toolkit or GPU’s that are failing. I have looked at all /usr/local/cuda../ folders and they all point to the same versions.</p>
<p>The only difference I can notice is that /usr/local/cuda../ has:</p>
<p>“nvidia_driver” : {<br>
“name” : “NVIDIA Linux Driver”,<br>
“version” : “570.124.06”<br>
},</p>
<p>and nvidia-smi has:</p>
<p>NVIDIA-SMI 570.133.20             Driver Version: 570.133.20</p>
<p>If you have any insight, it will be greatly appreciated.</p>

ptrblck: <aside class="quote no-group quote-modified" data-username="tovtyvold" data-post="12" data-topic="221604">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/bbce88/48.png" class="avatar"> tovtyvold:</div>
<blockquote>
<p>Regardless, the output samples give the following output during building:</p>
<p><code>nvcc fatal : Unsupported gpu architecture ‘compute_110’</code></p>
</blockquote>
</aside>
<p>This error is unrelated to the one you are seeing in this thread (losing access to your GPUs) and is caused by a mismatch in GPU architectures you are trying to build in the CUDA samples and your locally installed CUDA toolkit, which is too old and does not support <code>sm_110</code> yet (CUDA 13+ is needed).</p>
<p>As a simple workaround, remove <code>110</code> (and any other unwanted architectures) from <a href="https://github.com/NVIDIA/cuda-samples/blob/3f1c50965017932fc81e6d94a3fc9e04c105b312/CMakeLists.txt#L15">here</a>.</p>

tovtyvold: <p>I removed the unwanted architectures from the CMakeLists.txt files in the projects and built using:</p>
<p>$make -j$(nproc) –ignore-errors</p>
<p>since there were a lot of errors, type mismatches and such, and after that i ran:</p>
<p>python3 run_tests.py –parallel 4</p>
<p>which found around 77 executeables I think, and these were the ones that failed:<br>
Failed runs (26):<br>
update.sample : Failed (code 1)<br>
pre-merge-commit.sample : Failed (code 2)<br>
push-to-checkout.sample : Failed (code 1)<br>
pre-applypatch.sample : Failed (code 2)<br>
applypatch-msg.sample : Failed (code 2)<br>
CMakeDetermineCompilerABI_C.bin : Failed (code 49)<br>
CMakeDetermineCompilerABI_CXX.bin : Failed (code 49)<br>
a.out : Failed (code 212)<br>
a.out : Failed (code 212)<br>
a.out : Failed (code 190)<br>
fsmonitor-watchman.sample : Failed (code 255)<br>
streamOrderedAllocationP2P : Failed (code 2)<br>
ptxgen : Failed (code 16)<br>
cuda_f_1.yuv : Error: [Errno 8] Exec format error: ‘./cuda_f_1.yuv’ (code -1)<br>
cuda_yuv_f_1.yuv : Error: [Errno 8] Exec format error: ‘./cuda_yuv_f_1.yuv’ (code -1)<br>
cuda_yuv_f_2.yuv : Error: [Errno 8] Exec format error: ‘./cuda_yuv_f_2.yuv’ (code -1)<br>
CMakeDetermineCompilerABI_C.bin : Failed (code 49)<br>
CMakeDetermineCompilerABI_CXX.bin : Failed (code 49)<br>
a.out : Failed (code 212)<br>
a.out : Failed (code 212)<br>
a.out : Failed (code 190)<br>
build.bat : Error: [Errno 8] Exec format error: ‘./build.bat’ (code -1)<br>
build.sh : Failed (code 2)<br>
shaders.hlsl : Error: [Errno 8] Exec format error: ‘./shaders.hlsl’ (code -1)<br>
stdafx.cpp : Error: [Errno 8] Exec format error: ‘./stdafx.cpp’ (code -1)<br>
pre-push.sample : Timeout (code -1)</p>
<p>I do not know what to make of this. Since things are failing during building, it is unclear to me what is supposed to happen. Mind you I do not have experience with c++ so the depth of the errors are unclear. Also, looking at the APM_ output for the ones that failed, there is nothing that really stands out, most are empty.</p>