Cannot find GPU on A100 in Singularity

charliezchen: <p>After extensive research and consulting of Chat-GPT (which is not quite helpful in my opinion by the way), I still cannot resolve the problem, so here I am.</p>
<p>I am working on a HPC cluster and the environment is configured in Singularity. I am able to use the GPU normally when I requested a RTX8000, but when I requested an A100, the torch cannot find the GPU anymore. Online posts basically tell me to install the torch version with cuda correctly, but my concern here is that since the environment is in Singularity, I would want it to be agnostic to the GPU architecture. A straightforward way to make it work on A100 is to simply reinstall the torch or create a new conda env for the new GPU, but I don’t think this solution is elegant (this loses the significance of using singularity afterall). The driver versions for RTX8000 and A100 are the same, so I am guessing maybe using the same environment on two different architecture may be possible.</p>
<p>Some attempts I have tried:</p>
<ol>
<li>
<p>Using ldd to find where does the dependency of the dynamic library break<br>
To my surprise, the dependencies of the <code>libtorch.so</code> are primarily in the python site-packages. I thought it should be linked to the system cuda .so files. I added the cuda path on the machine to the LD_LIBRARY_PATH before I initiate the singularity container. The <code>libcublas.so</code> is now linked to the so file in <code>/usr/local/cuda-11.6/lib64/</code>, but still it doesn’t work. (BTW, in RTX8000, it seems that only libcublas are linked to the /usr/local path)</p>
</li>
<li>
</li>
</ol>
<p>print(torch.cuda.is_available())<br>
print(torch.backends.cudnn.enabled)<br>
torch.rand(1, device=“cuda”)</p>
<p>False<br>
True<br>
Runtime Error: GPU not found (paraphrased)</p>
<ol start="3">
<li>torch.utils.collect_env</li>
</ol>
<p>====</p>
<p>PyTorch version: 1.12.0+cu116<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.6<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 20.04.4 LTS (x86_64)<br>
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0<br>
Clang version: Could not collect<br>
CMake version: version 3.16.3<br>
Libc version: glibc-2.31</p>
<p>Python version: 3.8.13 (default, Mar 28 2022, 11:38:47)  [GCC 7.5.0] (64-bit runtime)<br>
Python platform: Linux-4.18.0-305.28.1.el8_4.x86_64-x86_64-with-glibc2.17<br>
Is CUDA available: False<br>
CUDA runtime version: 11.6.124<br>
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-80GB<br>
Nvidia driver version: 510.47.03<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.0<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.0<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.22.4<br>
[pip3] torch==1.12.0+cu116<br>
[pip3] torchaudio==0.12.0+cu116<br>
[pip3] torchmetrics==0.9.3<br>
[pip3] torchvision==0.13.0+cu116<br>
[conda] numpy                     1.22.4                   pypi_0    pypi<br>
[conda] torch                     1.12.0+cu116             pypi_0    pypi<br>
[conda] torchaudio                0.12.0+cu116             pypi_0    pypi<br>
[conda] torchmetrics              0.9.3                    pypi_0    pypi<br>
[conda] torchvision               0.13.0+cu116             pypi_0    pypi</p>
<p>====</p>
<p>I am so confused by the situation and resorted to use RTX8000 for now. I would be really grateful if anyone can provide me some insights on how to make this work or how does torch cuda work under the hood (like how does torch find the driver code for the GPU, I assume it is dynamic library linkage, but I am not quite familiar with this). If I miss any important information for this issue, please kindly let me know so I can supplement them.</p>

ptrblck: <aside class="quote no-group" data-username="charliezchen" data-post="1" data-topic="172936">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/charliezchen/48/57509_2.png" class="avatar"> charliezchen:</div>
<blockquote>
<p>The driver versions for RTX8000 and A100 are the same, so I am guessing maybe using the same environment on two different architecture may be possible.</p>
</blockquote>
</aside>
<p>No, that’s not the case. The PyTorch binaries shipping with CUDA 11.8 support compute capabilities 3.7 - 9.0 (the binaries with CUDA 11.7 do not support sm_90 [Hopper architecture]).</p>
<aside class="quote no-group" data-username="charliezchen" data-post="1" data-topic="172936">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/charliezchen/48/57509_2.png" class="avatar"> charliezchen:</div>
<blockquote>
<p>To my surprise, the dependencies of the <code>libtorch.so</code> are primarily in the python site-packages. I thought it should be linked to the system cuda .so files.</p>
</blockquote>
</aside>
<p>This is expected since the pip wheels and conda binaries ship with their own CUDA dependencies. The CUDA runtime, cuDNN, NCCL, and other libraries will be installed in the specified versions. Your locally installed CUDA toolkit is used to build custom CUDA extensions or PyTorch from source.</p>
<aside class="quote no-group" data-username="charliezchen" data-post="1" data-topic="172936">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/charliezchen/48/57509_2.png" class="avatar"> charliezchen:</div>
<blockquote>
<p>The <code>libcublas.so</code> is now linked to the so file in <code>/usr/local/cuda-11.6/lib64/</code>, but still it doesn’t work.</p>
</blockquote>
</aside>
<p>This sounds wrong since you are now trying to mix your system libraries into a pip wheel or conda binaries, which might have been built with another CUDA toolkit.</p>
<p>Since <code>print(torch.cuda.is_available())</code> returns <code>False</code> it points to a CUDA init error, which might be caused by a broken driver installation or maybe you are just masking the GPU via <code>CUDA_VISIBLE_DEVICES</code>, which was the case recently for another user running into the same issue.</p>

charliezchen: <p>Hi ptrblck,</p>
<p>The problem is solved (probably by the hpc administrator). Thanks for your response!</p>
<p>Best,</p>

ptrblck: <p>Good to hear it was solved! Do you know what the issue was and what exactly fixed it?</p>

charliezchen: <p>Unfortunately, I am not sure what solves the problem. I ran the collect_env again, and the only difference is that <code>is_cuda_available</code> is true now. It’s rather strange.</p>

Kiran_Beethoju: <p>I have solved this issue by updating the CUDA device info in commandline (terminal)</p>
<p><code>export CUDA_VISIBLE_DEVICES=0</code></p>
<p>If you are using latest cuda 12.4 in ubuntu<br>
If your GPU machine is A100</p>
<p>This will help you to resolve.</p>