PyTorch with CUDA 11 compatibility

Santhosh_Kumar1: <p>Recently, I installed a ubuntu 20.04 on my system. Since it was a fresh install I decided to upgrade all the software to the latest version. So, Installed <code>Nividia driver 450.51.05</code> version and <code>CUDA 11.0</code> version. To my surprise, Pytorch for CUDA 11 has not yet been rolled out.</p>
<p>My question is, should I downgrade the CUDA package to  10.2 or go with PyTorch built for CUDA 10.2 without downgrading CUDA itself?. Also, can anyone explain why PyTorch is built differently for various CUDA versions and what changes CUDA between versions?</p>

ptrblck: <p>As explained <a href="https://discuss.pytorch.org/t/install-pytorch-with-cuda-11/89219/4">here</a>, the binaries are not built yet with CUDA11. However, the initial CUDA11 enablement PRs are already merged, so that you could install from source using CUDA11.</p>
<p>If you want to use the binaries, you would have to stick to 10.2 for now.</p>

yueqi: <p>You probably don’t need to downgrade the CUDA 11 installed in your system. As explained <a href="https://discuss.pytorch.org/t/install-pytorch-gpu-with-pre-installed-cuda-and-cudnn/70808">here</a>,  <code>conda install pytorch torchvision cudatoolkit=10.2 -c pytorch</code>  will install CUDA 10.2 and cudnn binaries within the Conda environment, so the system-installed CUDA 11 will not be used at all.</p>
<p>I recently installed ubuntu 20.04 and Nvidia driver 450. It took me a while to realize that I didn’t have to build pytorch from source just because I have CUDA 11 in my system.</p>

rwightman: <p>Also, if you do actually want to try CUDA 11, easiest way is to make sure you have a sufficiently new driver and run the PyTorch NGC docker container. The latest 20.06 container has PyTorch 1.6, CUDA 11, and cuDNN 8, unfortunately cuDNN is an release candidate with some fairly significant performance regressions right now, not always the best idea to be bleeding edge <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Yuheng_Zhi: <p>I was trying to build from source following the steps listed <a href="https://github.com/pytorch/pytorch/#from-source" rel="nofollow noopener">here</a> with cuda 11.0 and a GTX 680 graphics card. But it stopped building due to an error:<br>
<code>nvcc fatal   : Unsupported gpu architecture 'compute_30'</code><br>
Would you mind giving me a bit advice on how to work around this? Much appreciated.</p>

ptrblck: <p>There is unfortunately no workaround for this, as compute capability 3.0 and 3.2 were dropped in CUDA11 and 3.5, 3.7, and 5.0 were deprecated (<a href="https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#deprecated-features">release notes</a>).</p>

AzimuthLI: <p>Life Saver! Bravo! Thanks!</p>

arogozhnikov: <p>Any progress?</p>
<p>Nvidia ships ubuntu 20 dockers only with Cuda 11, and it’s already more than half a year old.</p>

ptrblck: <p>You could build from source with CUDA11 (+cudnn8), use the NGC container, or the nightly binaries.<br>
Once 1.7 is code frozen, the stable binaries should be released.</p>

kepler62f: <p>Given above discussion, I’m surprised I was able to use PyTorch version 1.6 while my ubuntu20 host has CUDA version 11. For my setup, I used pytorch in a docker container using python3.8 base image and I pip installed pytorch 1.6 and torchvision 0.7. I did not even install any other cudatoolkit version. To access gpu from container I had nvidia-docker2 container toolkit package on the host. torch.cuda.is_available() returned true.</p>

ptrblck: <p>The conda binaries and pip wheels ship with their CUDA (cudnn, NCCL, etc.) runtime, so you don’t need a local CUDA installation to use native PyTorch operations.<br>
However, you would have to install a matching CUDA version, if you want to build PyTorch from source or build custom CUDA extensions.</p>

Ademord: <p>is there any update on this, i have torch ‘1.9.0+cu102’ but my system is cuda11.3</p>

ptrblck: <p>I’m not sure what kind of update you are looking for, but the binaries support <code>CUDA11</code> since PyTorch <code>1.7.0</code> I believe. You can select the corresponding CUDA runtime version by selecting the desired box <a href="https://pytorch.org/get-started/locally/">here</a>.</p>

nullgeppetto: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> What can be done for CUDA 11.4? Are binaries available? Could they be installed in a venv with pip? Many thanks!</p>

ptrblck: <p>The nightlies can be used with CUDA11.3. For CUDA11.4 we need to refactor the build pipeline a bit as the <code>cudatoolkit</code> binary was changed and we decided to work on it after the PyTorch 1.10.0 release.<br>
In the meantime you can of course build from source using CUDA 11.4.</p>

nullgeppetto: <p>Many thanks for your prompt reply!</p>

addisonklinke: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Is there a release roadmap anywhere that details when the PyTorch team is looking to add or deprecate support for specific CUDA versions? For instance, the current installation <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">guide</a> shows 10.2 as the default and 11.3 as an option (with the <code>+cu113</code> suffix for pip). I am wondering</p>
<ul>
<li>When will 11.x become the default without having to specify explicitly?</li>
<li>How much longer will 10.x be supported?</li>
<li>Is there an estimate for when &gt;= 12.x support will be added?</li>
</ul>
<p>Thank you for any references you can provide! This would be helpful for myself and other ML Engineers when choosing appropriate hardware</p>

ptrblck: <ol>
<li>
<p>We are working on making CUDA11 the default, but due to the large size of CUDA and cuDNN, it’s not an easy task. As you might have seen the size difference between the wheels shipped with CUDA10.2 vs. 11.3 is huge and we are checking different approaches right now.</p>
</li>
<li>
<p>See point 1. We hope to be able to come up with a valid approach so that we can release with one setup.</p>
</li>
<li>
<p>Once CUDA12 is out, we’ll add it (first as a source build option, then in the wheels) <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>
</li>
</ol>

GiantRice: <p>hi, <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I have some confusion about CUDA compatibility.</p>
<p>We have a NVIDIA A10 GPU,<br>
and nvidia-smi  shows ( NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4)</p>
<p>We install pytorch with pip method. below wheel pkgs<br>
<code>torch-1.3.0%2Bcu100-cp37-cp37m-linux_x86_64.whl</code><br>
<code>torchvision-0.4.1%2Bcu100-cp37-cp37m-linux_x86_64.whl</code></p>
<p>should torch-1.3.0+cu100 be compatible with my gpu driver version?</p>
<p>When we run below python code, it takes about  too long time(more than 8 minutes) to finish. we find cpu 100% using top command.  So why does it cost so long time?</p>
<pre><code class="lang-auto">import torch
torch.Tensor([1,2]).cuda()
</code></pre>
<p>And with some other torch code like <code>torch.relu_(input)</code>, it encounters errors ?<br>
<code>RuntimeError: CUDA error: no kernel image is available for execution on the device</code></p>

ptrblck: <aside class="quote no-group" data-username="GiantRice" data-post="20" data-topic="89254">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/giantrice/48/43421_2.png" class="avatar"> GiantRice:</div>
<blockquote>
<p>should torch-1.3.0+cu100 be compatible with my gpu driver version?</p>
</blockquote>
</aside>
<p>The used CUDA runtime would be compatible. However, you are using an Ampere GPU which needs CUDA&gt;=11.0.<br>
The CUDA 11 runtime landed in PyTorch 1.7, so you would need to update the PyTorch pip wheels to any version after 1.7 (I would recommend to use the latest one) with the CUDA11 runtime (the current 1.10.0 pip wheels use CUDA11.3).</p>