PyTorch detect GPU but can't run model on GPU - A6000

werdas34: <p>I am working with a paperspace vm with an A6000 GPU and have a transformers/pytorch model there.<br>
I can’t get this to run on this GPU but it runs on the CPU.<br>
PyTorch must have version 1.4.<br>
The following is the setup of the VM and sample code.<br>
Interestingly, the same model with the same settings (except nvcc, because no toolkit) runs fine on a P9000 from paperspace.</p>
<p>Anybody have an idea how I can get the model to run on the A6000 anyway?</p>
<pre><code class="lang-python">import transformers, torch
print(transformers.__version__)
print(torch.__version__)
3.2.0
1.4.0+cu92

torch.cuda.is_available()
True

torch.cuda.device_count()
1

torch.cuda.current_device()
0

torch.cuda.device(0)
&lt;torch.cuda.device at 0x7f679aac13d0&gt;

torch.cuda.get_device_name(0)
'NVIDIA RTX A6000'

torch.version.cuda
'9.2'

nvidia-smi
Thu Aug 11 19:21:54 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA RTX A6000    On   | 00000000:00:05.0 Off |                  Off |
| 30%   35C    P8    18W / 300W |      5MiB / 49140MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1425      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+

gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

g++ --version
g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.4 LTS
Release:	20.04
Codename:	focal

x = torch.rand(5, 3, device="cpu")
print(x)
tensor([[0.8225, 0.6572, 0.9615],
        [0.5285, 0.8547, 0.3622],
        [0.8873, 0.2297, 0.5951],
        [0.1628, 0.1683, 0.3286],
        [0.5517, 0.0598, 0.7807]])

x = torch.rand(5, 3, device="cuda:0")
print(x)
# infinity-loop, does not terminate, no error message
</code></pre>

ptrblck: <aside class="quote no-group" data-username="werdas34" data-post="1" data-topic="158958">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/c37758/48.png" class="avatar"> werdas34:</div>
<blockquote>
<p>PyTorch must have version 1.4.</p>
</blockquote>
</aside>
<p>That won’t work on your A6000, as you need CUDA=11.x for your Ampere GPU and thus cannot use the CUDA 9.2 runtime.</p>

werdas34: <p>Thank you for the quick reply. That explains a lot, of course.</p>
<p>I need a GPU with more than 24GB GPU RAM. Do you have any idea where I can find a cloud solution?</p>
<p><a href="https://docs.paperspace.com/core/compute/machine-types/" rel="noopener nofollow ugc">Paperspace</a> also offers a V100. Would this work? I’m guessing A100 won’t work either.</p>

ptrblck: <p>CUDA 9.2 would support compute capability 7.0, so Volta should work.<br>
A100 will not work with CUDA 9.2 as it’s also an Ampere GPU (sm_80) and thus needs CUDA 11.</p>
<p>In any case, your best bet would be to update to the latest PyTorch release and use a current CUDA runtime.</p>

werdas34: <p>Thanks for the answer.</p>
<p>I would like to update to the latest version. I am working on a model that was released last year with PyTorch 1.4. Updating it dry and dirty did not work.<br>
And I am not an expert in PyTorch.</p>
<p>Edit:<br>
Which pytorch version would I need to have at least for the A6000?</p>

ptrblck: <p>PyTorch 1.7.0 should have introduced CUDA 11, but note that this version is already quite old as it was released in late Oct 2020 so I would still recommend to try to update your repository to be compatible with the latest stable release.</p>