How to check torch gpu compatibility without initializing CUDA?

George3d6: <p>Older GPUs don’t seem to support torch in spite of recent cuda versions.</p>
<p>In my case the crash has the following error:</p>
<pre><code class="lang-auto">/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/cuda/__init__.py:83: UserWarning: 
    Found GPU%d %s which is of cuda capability %d.%d.
    PyTorch no longer supports this GPU because it is too old.
    The minimum cuda capability supported by this library is %d.%d.
    
  warnings.warn(old_gpu_warn.format(d, name, major, minor, min_arch // 10, min_arch % 10))
WARNING:lightwood-16979:Exception: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1. when training model: &lt;lightwood.model.neural.Neural object at 0x7f9c34df1e80&gt;
Process LearnProcess-1:13:
Traceback (most recent call last):
  File "/home/maxs/dev/mdb/venv38/sources/lightwood/lightwood/model/helpers/default_net.py", line 59, in forward
    output = self.net(input)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 96, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/functional.py", line 1847, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>
<p>This happens in spite of:</p>
<pre><code class="lang-auto">assert torch.cuda.is_available() == True
torch.version.cuda == '10.2'
</code></pre>
<p>How can I check for an older GPU that doesn’t support torch <em>without</em> actually try/catching a tensor-to-gpu transfer? The transfer initializes cuda, which wastes like 2GB of memory, something I can’t afford since I’d be running this check in dozens of processes, all of which would then waste 2GB of memory extra due to the initialization.</p>

George3d6: <p>Since the answer here might not be very torch-related and instead implicate usage of e.g. nvidia specific tools I also asked here: <a href="https://stackoverflow.com/questions/68650792/how-to-check-torch-gpu-compatibility-without-initializing-cuda" class="inline-onebox" rel="noopener nofollow ugc">python - How to check torch gpu compatibility without initializing CUDA? - Stack Overflow</a> | Putting in the link in case anyone stumbles upon this question later and finds the SO answers useful.</p>

el_youssfi_azeddine: <p>I think the latest cuda version vailable is 11.3, use the command provided in pytorch installation guide <a href="https://pytorch.org" rel="noopener nofollow ugc">https://pytorch.org</a><br>
It installs automatically pytorch cuda compatible.<br>
Pardon if I misunderstood your request! <img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=10" title=":upside_down_face:" class="emoji" alt=":upside_down_face:"></p>

Manuel_Alejandro_Dia: <p>It seems like you are correct assuming:</p>
<blockquote>
<p>Since the answer here might not be very torch-related and instead implicate usage of e.g. nvidia specific tools</p>
</blockquote>
<p>According to <a href="https://discuss.pytorch.org/t/solved-pytorch-no-longer-supports-this-gpu-because-it-is-too-old/15444/10">this comment</a> there is a relationship between the CUDA version and the deprecated compute capabilities you won’t be able to use. You can check the <a href="https://docs.nvidia.com/cuda/archive/10.2/cuda-toolkit-release-notes/index.html#deprecated-features" rel="noopener nofollow ugc">CUDA 10.2 Toolkit Docs</a> where it says which <a href="https://developer.nvidia.com/cuda-gpus" rel="noopener nofollow ugc">compute capabilities</a> are deprecated.</p>

George3d6: <p>I am more so referring to doing this check on someone else machine, that already has torch installed.</p>
<p>I.e. for a package that a user installs where I have to check “given this environment, should I run this with cuda or on the cpu?”</p>

Manuel_Alejandro_Dia: <p>It might be not be the best solution, but you could do a look-up table where you do a relationship between CUDA versions and deprecated GPUs.<br>
Then, using <code>torch.cuda.get_device_name(torch.cuda.current_device())</code> you could check if the code should be executed on GPU or CPU.</p>

KFrank: <p>Hi George!</p>
<aside class="quote no-group" data-username="George3d6" data-post="5" data-topic="128528" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/george3d6/48/21758_2.png" class="avatar"> George3d6:</div>
<blockquote>
<p>I am more so referring to doing this check on someone else machine, that already has torch installed.</p>
</blockquote>
</aside>
<p>As far as I know, the only airtight way to check cuda / gpu compatibility<br>
is <code>torch.cuda.is_available()</code> (and to be completely sure, actually<br>
perform a tensor operation on the gpu).  That’s what I do on my own<br>
machines (but once I check a that a given version of pytorch works with<br>
my gpu, I don’t have to keep doing it).</p>
<p>You want to check somebody else’s pytorch with somebody else’s gpu,<br>
so I would say it’s doubly important to actually run the gpu.</p>
<p>I would run a separate python process that runs a simple gpu test<br>
script before running your “real” program.  (It could store its result in<br>
a text file or environment variable or simply inform the user.)  When<br>
that process exits, it will release any cuda overhead.</p>
<p>If you want to get fancy, you could have your “real” program spawn<br>
your test script in a separate process, and proceed with or without<br>
the gpu depending on the test script’s results.  Again, when the<br>
spawned process exits, its cuda overhead will be released.</p>
<p>Best.</p>
<p>K. Frank</p>

George3d6: <p>Based on the code in <code>torch.cuda.__init__</code> that was actually throwing the error the following check seems to work:</p>
<pre><code class="lang-auto">import torch
from torch.cuda import device_count, get_device_capability


def is_cuda_compatible():
    if torch.version.cuda is not None:
        compatible_device_count = 0
        for d in range(device_count()):
            capability = get_device_capability(d)
            major = capability[0]
            minor = capability[1]
            current_arch = major * 10 + minor
            min_arch = min((int(arch.split("_")[1]) for arch in torch.cuda.get_arch_list()), default=35)
            if (not current_arch &lt; min_arch
                    or not torch._C._cuda_getCompiledVersion() &lt;= 9000
                    or not (major &gt;= 7 and minor &gt;= 5)):
                compatible_device_count += 1

        return True
    return False
</code></pre>
<p>Not sure if it’s 100% correct but putting it out here for feedback and in case anybody else needs it. Will be PRing into torch itself later since it seems like the kind of functionality it ought to have.</p>

George3d6: <p>Based on the code in <code>torch.cuda.__init__</code> that was actually throwing the error the following check seems to work:</p>
<pre><code class="lang-auto">import torch
from torch.cuda import device_count, get_device_capability

def is_cuda_compatible():
    compatible_device_count = 0
    if torch.version.cuda is not None:
        for d in range(device_count()):
            capability = get_device_capability(d)
            major = capability[0]
            minor = capability[1]
            current_arch = major * 10 + minor
            min_arch = min((int(arch.split("_")[1]) for arch in torch.cuda.get_arch_list()), default=35)
            if (not current_arch &lt; min_arch
                    and not torch._C._cuda_getCompiledVersion() &lt;= 9000):
                compatible_device_count += 1

    if compatible_device_count &gt; 0:
        return True
    return False
</code></pre>
<p>Not sure if it’s 100% correct but putting it out here for feedback and in case anybody else needs it. Will be PRing into torch itself later since it seems like the kind of functionality it ought to have.</p>