NVIDIA Quadro 4000 too old

username: aclifton314
cooked: <p><strong>SYSTEM</strong><br>
OS: Ubuntu 18.04<br>
Python version: 3.6.8<br>
Torch version: 1.0.0<br>
GPU: Quadro 4000<br>
Cuda: 8.0<br>
NVIDIA Driver: 390</p>
<p>I am using the transformers framework to retrain a NLP model. Ultimately, I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at /pytorch/aten/src/THC/generic/THCTensorMath.cu:238
</code></pre>
<p>From what I can gather, pytorch finds the GPU, but says it is too old.</p>
<pre><code class="lang-auto"> Found GPU0 Quadro 4000 which is of cuda capability 2.0.
    PyTorch no longer supports this GPU because it is too old.
</code></pre>
<p>However, that is a warning. What I read was that the gpu is no longer supported by pytorch (despite cuda working with the gpu and this version of pytorch is recommended for cuda 8.0). I was wondering if there were a remedy to this problem (other than getting a new gpu)? I have read on here that it might be possible to build pytorch from source, but I have no idea how to do that.</p>
<p>Any guidance is much appreciated. Thanks in advance!</p>

username: KFrank
cooked: <p>Hi Alex!</p>
<p>I don’t have much of an answer for you, but I can share my experience.</p>
<aside class="quote no-group quote-modified" data-username="aclifton314" data-post="1" data-topic="61020" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/9fc348/48.png" class="avatar"> aclifton314:</div>
<blockquote>
<p>From what I can gather, pytorch finds the GPU, but says it is too old.</p>
<pre><code class="lang-auto"> Found GPU0 Quadro 4000 which is of cuda capability 2.0.
    PyTorch no longer supports this GPU because it is too old.
</code></pre>
<p>I was wondering if there were a remedy to this problem (other than getting a new gpu)?</p>
</blockquote>
</aside>
<p>I found myself in a similar situation as discussed in this thread:</p>
<aside class="quote quote-modified" data-post="1" data-topic="51263">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pre-built-pytorch-for-cuda-compute-capability-3-0-on-windows/51263">Pre-built pytorch for CUDA [compute capability] 3.0 on windows?</a> 
  </div>
  <blockquote>
    Hi Forum! 
Would anybody know of a pre-built pytorch windows / CUDA 3.0 
version?  (It’s windows 10, if that matters.) 
I’m aware that pytorch no longer formally supports older CUDA 
versions, but I have seen older pre-built packages floating 
around on the internet – just not this configuration. 
(I will be installing pytorch on an older laptop with a CUDA 3.0 
quadro k1100m graphics card.  I’ll just be monkeying around, 
so I don’t need the gpu, but I’d like to monkey around with 
the gpu, as …
  </blockquote>
</aside>

<p>The short story is that with some help from the experts here, I was<br>
able to find a pre-built pytorch that supported my creaky gpu.  But<br>
this came at the cost of running the old, 0.3.0 version of pytorch.</p>
<p>Since (if I understand you correctly) you’re looking for compute<br>
capability 2.0, the outlook for you is probably bleaker still.  But<br>
maybe there is something out there that will work for you.  (You<br>
might take a look in the links suggested in the aforementioned<br>
thread.)</p>
<aside class="quote no-group">
<blockquote>
<p>I have read on here that it might be possible to build pytorch from source, but I have no idea how to do that.</p>
</blockquote>
</aside>
<p>Building pytorch myself was also suggested as a possible solution,<br>
but I elected not to try it.  I have no idea how easy or hard it would be,<br>
but my expectation is that doing so would be a substantive project<br>
in its own right.  (Maybe it would be easier than I think, or maybe you<br>
are more courageous than I …).</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: aclifton314
cooked: <p>Hi KFrank,</p>
<p>Thanks for your message. Let me elaborate more on what I have done.</p>
<p>For my particular gpu, I was able to get the gpu and cuda to work together (as evident from calling the deviceQuery binary installed whenever I installed cuda). This gpu, which is a cuda capability 2.0, works with cuda 8.</p>
<p>The most recent version of pytorch will not work with this gpu. So I went to the pytorch website and looked for previous versions of pytorch that will work with cuda 8 (those versions can be found <a href="https://pytorch.org/get-started/previous-versions/" rel="nofollow noopener">here</a>) and that turned out to be version 1.0.0. I got that installed with</p>
<pre><code class="lang-auto">pip install torch==1.0.0 torchvision==0.2.1
</code></pre>
<p>then tested pytorch to see if it could find and identify my gpu:</p>
<pre><code class="lang-auto">Python 3.6.8 (default, Oct  7 2019, 12:59:55)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.current_device()
/home/user/.local/lib/python3.6/site-packages/torch/cuda/__init__.py:117: UserWarning:
    Found GPU0 Quadro 4000 which is of cuda capability 2.0.
    PyTorch no longer supports this GPU because it is too old.

  warnings.warn(old_gpu_warn % (d, name, major, capability[1]))
0
&gt;&gt;&gt; torch.cuda.device(0)
&lt;torch.cuda.device object at 0x7fb833751d30&gt;
&gt;&gt;&gt; torch.cuda.device_count()
1
&gt;&gt;&gt; torch.cuda.get_device_name(0)
'Quadro 4000'
&gt;&gt;&gt; torch.cuda.is_available()
True
</code></pre>
<p>At this point, I am assuming that cuda, my gpu, and this older version of pytorch are all playing nicely. However, when I go to train my model I get that RuntimeError I posted originally (copied below):</p>
<pre><code class="lang-auto">RuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at /pytorch/aten/src/THC/generic/THCTensorMath.cu:238
</code></pre>
<p>Here is the full output of the error, if that would be more useful:</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCTensorMath.cu line=238 error=48 : no kernel image is available for execution on the device
Traceback (most recent call last):
  File "run_lm_finetuning.py", line 548, in &lt;module&gt;
    main()
  File "run_lm_finetuning.py", line 500, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 206, in train
    output_device=args.local_rank)
  File "/home/user/.local/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 215, in __init__
    self.broadcast_bucket_size)
  File "/home/user/.local/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 377, in _dist_broadcast_coalesced
    dist._dist_broadcast_coalesced(self.process_group, tensors, buffer_size, False)
RuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at /pytorch/aten/src/THC/generic/THCTensorMath.cu:238

</code></pre>
<p>EDIT:<br>
One last bit of information that might be useful. I am calling the python script as follows:</p>
<pre><code class="lang-auto">python -m torch.distributed.launch run_lm_finetuning.py
</code></pre>

username: KFrank
cooked: <p>Hello Alex!</p>
<aside class="quote no-group quote-modified" data-username="aclifton314" data-post="3" data-topic="61020" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/9fc348/48.png" class="avatar"> aclifton314:</div>
<blockquote>
<p>For my particular gpu, I was able to get the gpu and cuda to work together (as evident from calling the deviceQuery binary installed whenever I installed cuda). This gpu, which is a cuda capability 2.0, works with cuda 8.</p>
<pre><code class="lang-auto">pip install torch==1.0.0 torchvision==0.2.1
</code></pre>
<p>then tested pytorch to see if it could find and identify my gpu:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.cuda.current_device()
/home/user/.local/lib/python3.6/site-packages/torch/cuda/__init__.py:117: UserWarning:
    Found GPU0 Quadro 4000 which is of cuda capability 2.0.
    PyTorch no longer supports this GPU because it is too old.
</code></pre>
<p>At this point, I am assuming that cuda, my gpu, and this older version of pytorch are all playing nicely. However, when I go to train my model I get that RuntimeError I posted originally (copied below):</p>
<pre><code class="lang-auto">RuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at /pytorch/aten/src/THC/generic/THCTensorMath.cu:238
</code></pre>
</blockquote>
</aside>
<p>I don’t remember in detail how things unfolded when I got an old<br>
build of pytorch to work with my old gpu.  However, my understanding<br>
of this (which could well be wrong) is that it is perfectly possible for<br>
you to have your cuda version / cuda driver work properly with<br>
your gpu, and have your version of pytorch work properly with your<br>
cuda driver, but have pytorch not work with the specific compute<br>
capability of your gpu.</p>
<p>I <em>believe</em> that a given cuda version / cuda driver typically works with<br>
a range of compute capabilities.</p>
<p>In your case, your version of pytorch explicitly warns you that it needs<br>
a higher compute capability.</p>
<p>So my interpretation is that yes, in a sense, pytorch and cuda and your<br>
gpu are working together, and pytorch correctly recognizes your gpu.<br>
But your version of pytorch knows that it uses some post-capability-2.0<br>
gpu computation features, and warns you of this.  (You could argue<br>
that pytorch could simply say “no gpu” at this point, but it chooses to<br>
admit that there is a recognized gpu, but warns you it’s not good enough.)</p>
<p>Then when you actually try to use the gpu (or at least some specific<br>
“kernel” that uses an unsupported feature), you get the runtime error.</p>
<p>My recollection and understanding is that somewhere in the pytorch<br>
code is the required cuda compute capability, but that this has not<br>
be systematically recorded for the various pytorch builds.  So you can<br>
either try installing a build and see what it thinks of your cpu, or if<br>
you can somehow match up a build to the version of the source from<br>
which it was built, you could look in the code to check the required<br>
compute capability.</p>
<p>What I did when I wanted to use pytorch with my old gpu was to keep<br>
trying older and older archived pytorch builds until I found one that<br>
worked.</p>
<p>But the bottom line is <code>torch.cuda.is_available() == True</code> is not<br>
equivalent to “this pytorch works with this gpu.”</p>
<p>Have fun.</p>
<p>K. Frank</p>

username: aclifton314
cooked: <aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="61020">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>However, my understanding<br>
of this (which could well be wrong) is that it is perfectly possible for<br>
you to have your cuda version / cuda driver work properly with<br>
your gpu, and have your version of pytorch work properly with your<br>
cuda driver, but have pytorch not work with the specific compute<br>
capability of your gpu.</p>
</blockquote>
</aside>
<p>This is exactly what I was wondering about. I agree with your recommendation and will keep trying older versions of pytorch to see if I find one that will work with my gpu. Thank you for your detailed response!</p>
<p>Does anyone know where I can download older version of pytorch for linux?</p>