Torch::cuda::is_available() returns false

username: Deepali
cooked: <p>I have built <code>PyTorch v1.4.0</code> locally and I am trying the <code>C++</code> frontend.</p>
<p>I am seeing that the CPP API - <code>torch::cuda::is_available()</code> returns false.<br>
With the following snippet:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;

int main() 
{
    torch::Tensor a = torch::ones({ 2, 2 }, torch::requires_grad());
    torch::Tensor b = torch::randn({ 2, 2 });
    a.to(torch::kCUDA);    
    b.to(torch::kCUDA);   
    auto c = a + b;
}
</code></pre>
<p>I saw the complaint:<br>
<code> what():  CUDA error: CUDA driver version is insufficient for CUDA runtime version</code></p>
<p>However in python I don’t see any such issue:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; a = torch.tensor([1., 2.], device="cuda")
&gt;&gt;&gt; a
tensor([1., 2.], device='cuda:0')

</code></pre>
<p>What could be wrong here?</p>

username: 21fa417e3fb06e56040c
cooked: <p>When you try torch::cuda::is_available() return false?<br>
right?</p>
<p>Try modifying it like the code below.</p>
<pre><code class="lang-auto">    torch::Tensor a = torch::ones({ 2, 2 }, torch::requires_grad());
    torch::Tensor b = torch::randn({ 2, 2 });
    a = a.to(torch::kCUDA);    
    b = b.to(torch::kCUDA); 
</code></pre>

username: 21fa417e3fb06e56040c
cooked: <p>Also check your system.<br>
You can get the script and run it with:</p>
<pre><code class="lang-auto">wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
</code></pre>

username: Deepali
cooked: <p>Cuda information collected is:</p>
<pre><code class="lang-auto">Is CUDA available: Yes
CUDA runtime version: 10.2.89
Nvidia driver version: 440.33.01
</code></pre>

username: hartbx
cooked: <p>I work with <a class="mention" href="/u/deepali">@Deepali</a> and have been looking at this as well. I think I see what’s happening now…</p>
<p>The <code>libcuda.so</code> that’s being loaded at run time is the stub version that comes with CUDA Toolkit. That’s suitable for dynamic linking at build time, but isn’t actually functional.</p>
<pre><code class="lang-auto">$ LD_DEBUG=all ./libtest 2&gt;&amp;1 | grep 'object=.*libcuda[.]so'
     95995:     object=/opt/anaconda3/envs/test_env/lib64/stubs/libcuda.so.1 [0]
     95995:     object=/opt/anaconda3/envs/test_env/lib64/stubs/libcuda.so.1 [0] 
</code></pre>
<p>And <em>that</em> library is being found because of the <code>RPATH</code> in the built binary:</p>
<pre><code class="lang-auto">$ objdump -p libtest | grep RPATH
  RPATH                [...]:/opt/anaconda3/envs/test_env/lib64/stubs:[...]
</code></pre>
<p>Unfortunately <code>RPATH</code> takes precedence over <code>LD_LIBRARY_PATH</code> :</p>
<pre><code class="lang-auto">$ LD_LIBARY_PATH=/usr/lib64 ./libtest
device_count: 0
Is available? 0
</code></pre>
<p>But <code>LD_PRELOAD</code> would get around the problem:</p>
<pre><code class="lang-auto">$ LD_PRELOAD=/usr/lib64/libcuda.so.440.33.01 ./libtest
device_count: 4
Is available? 1
</code></pre>
<p>So I think the trick will be to find a way to prevent the CUDA Toolkit <code>stubs/</code> library directory from being added to <code>RPATH</code>.</p>
<p>I’m not sure whether it’s something specific to our environment (we’re building in a conda setup) that causes that, or if it would be something in the general libtorch stuff.</p>
<p>I tried just not installing the CUDA Toolkit development stuff (leaving only the runtime installed), but then the Torch cmake find_package stuff fails:</p>
<pre><code class="lang-auto">CUDA_TOOLKIT_ROOT_DIR not found or specified                                                                                                                                       
-- Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS)
</code></pre>
<p>I didn’t look into that yet, but assume it’s probably looking for <code>nvcc</code> to figure out where CUDA’s installed.</p>

username: hartbx
cooked: <p>I’ve also asked NVIDIA if they’ll consider updating the stub library’s <code>cuInit()</code> routine to return a well-defined error result (<code>CUDA_STUB_LIBRARY</code> or somesuch). That would make it immediately obvious what’s going on in cases like these.</p>

username: Shisho_Sama
cooked: <p>I’m facing the same issue on Windows 10.<br>
I’m using Pytorch 1.7.1cu101 and The drivers are up to date(i.e. 466.11), Pytorch detects the GPU and thus <code>torch.cuda.is_available()</code> returns True, however, its the libtorch that fails! and always returns false.</p>
<p>I have tested with older driver as well (457? I guess) to no avail!<br>
I also uninstalled all versions of cuda toolkits available on the system, hoping that might have had something to do with it, as they were in PATH and they might have caused a conflict with locally shipped dlls from libtorch libs, that wasnt the case either!<br>
So what could be the issue here?<br>
Here is the output of collect_env:</p>
<pre><code class="lang-cpp">PS C:\Users\user&gt; python collect_env.py
Collecting environment information...
PyTorch version: 1.7.0
Is debug build: True
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Enterprise
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.9.0

Python version: 3.7 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.1.105
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1080
Nvidia driver version: 466.11
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] numpydoc==0.9.2
[pip] torch==1.7.0
[pip] torchaudio==0.7.0
[pip] torchvision==0.8.1
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.0                      166
[conda] mkl-service               2.3.0            py37hb782905_0
[conda] mkl_fft                   1.0.15           py37h14836fe_0
[conda] mkl_random                1.1.0            py37h675688f_0
[conda] numpy                     1.18.1           py37h93ca92e_0
[conda] numpy-base                1.18.1           py37hc3f5095_1
[conda] numpydoc                  0.9.2                      py_0
[conda] torch                     1.7.0                    pypi_0    pypi
[conda] torchaudio                0.7.0                    pypi_0    pypi
[conda] torchvision               0.8.1                    pypi_0    pypi
</code></pre>
<p>Any help is greatly appreciated</p>

username: Shisho_Sama
cooked: <p>OK, found the culprit!<br>
It seems to be a known bug as stated <a href="https://github.com/pytorch/pytorch/issues/31611" rel="noopener nofollow ugc">here</a>, and the reason is the cuda.dll wont get loaded for some reason and thus you face this issue.<br>
To get around this issue, one way that was proposed was to add this :</p>
<pre><code class="lang-auto">/INCLUDE:"?warp_size@cuda@at@@YAHXZ"
</code></pre>
<p>to the linker flags. (in VS this means to add it to the linker/commands section)<br>
this solves the issue.</p>