How to get debug info from `torch.cuda.is_available()` about missing libs

username: t.g
cooked: <p>Hi,</p>
<p>There are already a ton of posts on this topic of " torch is not recognizing GPU backend".<br>
Each of the solutions mentioned is per case (such as upgrade or downgrade cuda or driver to some magic number for a specific torch version; see <a href="https://devtalk.nvidia.com/default/topic/1047223/torch-cuda-is_available-returns-false/" rel="nofollow noopener">1</a>, <a href="https://stackoverflow.com/questions/55717751/torch-cuda-is-available-keeps-switching-to-false" rel="nofollow noopener">2</a>, <a href="https://discuss.pytorch.org/t/why-torch-cuda-is-available-still-returned-false-even-i-installed-cuda-9-0-176-help/33364">3</a>, <a href="https://discuss.pytorch.org/t/torch-cuda-is-available-returns-false-nvidia-smi-is-working/20614">4</a>, <a href="https://github.com/pytorch/pytorch/issues/15612" rel="nofollow noopener">5</a>, <a href="https://discuss.pytorch.org/t/pytorch-0-4-get-cuda-version/19058/2">6</a> …), and none of those answers seem generic enough to address most people, so they keep coming with new versions of torch.</p>
<p>So, here is what I learned</p>
<p>Currently</p>
<pre><code class="lang-auto">$ python -c 'import torch; print(torch.cuda.is_available())'
False
</code></pre>
<p>But why? I got <code>nvidia-smi</code> showing GPU correctly.</p>
<p>One sure-shot way of fixing my cuda lib compatibility problems (not the desirable, though!)  is asking tensorflow</p>
<pre><code class="lang-auto">$ python -c 'import tensorflow as tf; print(tf.test.is_gpu_available())'

2019-11-18 22:56:28.982050: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:00:06.0
2019-11-18 22:56:28.986522: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
2019-11-18 22:56:28.989327: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory
2019-11-18 22:56:28.992141: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory
2019-11-18 22:56:28.994862: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory
2019-11-18 22:56:28.997474: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory
2019-11-18 22:56:29.000329: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory
2019-11-18 22:56:29.003020: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory
2019-11-18 22:56:29.003104: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2019-11-18 22:56:29.003232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-11-18 22:56:29.003284: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0
2019-11-18 22:56:29.003372: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N
False
</code></pre>
<p>TF precisely states <code>Could not load dynamic library &lt;namehere&gt;</code>, example <code>libcudnn.so.7</code> or <code>'libcublas.so.10.0'</code>  which is a crucial info to know what cuda libs and versions are missing.</p>
<p>If there is a way to get such debug info from torch? please let me know. (we would be pleased to not depend on TF to fix it).   <strong>Should the <code>torch.cuda.is_available()</code> have <code>debug=True</code> argument to print which of the missing libraries are causing it to return <code>False</code>?</strong></p>
<hr>
<p>With this, if we figure out the missing libs and version are:<br>
<code>Could not load dynamic library 'libcudnn.so.7';</code> or  <code>'libcublas.so.10.0'</code><br>
We can do</p>
<pre><code class="lang-auto">conda install cudnn=7 cudatoolkit=10.0 -c anaconda
</code></pre>
<p>then for sure torch recognizes GPU backend:</p>
<pre><code class="lang-auto">$ python -c 'import torch; print(torch.cuda.is_available())'
True
</code></pre>
<p>But we have to know first that the missing libs are <code>'libcudnn.so.7'</code> and <code>'libcublas.so.10.0'</code> to install <code>cudnn=7 cudatoolkit=10.0</code> (otherwise it goes to trial-error and magic numbers)</p>

username: t.g
cooked: <p>Also, another topic related to getting extra info from <code>debug=True</code>:</p>
<pre><code class="lang-auto">$ python -c 'import torch; print(torch.cuda.is_available(), torch.version.cuda)'
False 10.0.130
</code></pre>
<pre><code class="lang-auto">$ python -c 'import tensorflow as tf; print(tf.test.is_gpu_available())'
...
E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error
...
False
</code></pre>
<p>I guess <code>cuInit: CUDA_ERROR_UNKNOWN: unknown error</code> means I have to restart the machine. Knowing this errors would help as well.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can try to actually use a cuda element to see such errors: <code>torch.rand(1, device="cuda")</code>.<br>
Does that give you the informations you want?</p>

username: t.g
cooked: <p>Yes that helps. It prints useful info.  Thanks.<br>
Wish it was a documented feature!</p>
<p>My code branching has been</p>
<pre><code class="lang-auto">if torch.cuda.is_available():
    use cuda
else:
    use cpu, dont attempt to use of cuda
</code></pre>
<p>so I never had chance to see that message</p>