CUDA driver initialization failed

username: u_cult
cooked: <p>I use PyTorch NGC Container 23.05-py3. And use `nvidia-docker run -it --rm -v <a href="http://nvcr.io/nvidia/pytorch:23.05-py3" rel="noopener nofollow ugc">nvcr.io/nvidia/pytorch:23.05-py3</a> to launch it. It show that</p>
<pre><code class="lang-auto">
ERROR: The NVIDIA Driver is present, but CUDA failed to initialize.  GPU functionality will not be available.
   [[ Initialization error (error 3) ]]

</code></pre>
<p>I run <code>nvidia-smi</code>，it output</p>
<pre><code class="lang-auto">Wed Jun  7 12:17:34 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.41.03              Driver Version: 530.41.03    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-SXM4-80GB           Off| 00000000:40:00.0 Off |                   On |
| N/A   27C    P0               31W / 400W|      0MiB / 81920MiB |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A100-SXM4-80GB           Off| 00000000:B1:00.0 Off |                   On |
| N/A   26C    P0               36W / 400W|      0MiB / 81920MiB |     N/A      Default |
|                                         |                      |              Enabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| MIG devices:                                                                          |
+------------------+--------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |                   Memory-Usage |        Vol|      Shared           |
|      ID  ID  Dev |                     BAR1-Usage | SM     Unc| CE ENC DEC OFA JPG|
|                  |                                |        ECC|                       |
|==================+================================+===========+=======================|
|  No MIG devices found                                                                 |
+---------------------------------------------------------------------------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

</code></pre>
<p>I run <code>nvcc --version</code>,it show</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0
</code></pre>
<p>Besides, I run this contain in a lxc contain.</p>

username: ptrblck
cooked: <p>Try to disable MIG and check if any other bare metal CUDA application would work in your setup.</p>

username: u_cult
cooked: <p>It works!  I wonder where the problem is. Can I use the MIG later？</p>

username: u_cult
cooked: <p>It works! I wonder where the problem is. Can I use the MIG later？</p>

username: ptrblck
cooked: <p>Yes, you should be able to use MIG but would need to stick to the <a href="https://docs.nvidia.com/datacenter/tesla/mig-user-guide/">user guide</a> and create proper compute instances etc.</p>