PyTorch 1.6 - "Tesla T4 with CUDA capability sm_75 is not compatible"

sniklaus: <p>I installed PyTorch 1.6 with pip as follows.</p>
<pre><code class="lang-auto">pip install --no-cache-dir torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>However, PyTorch prints the following now.</p>
<pre><code class="lang-auto">Tesla T4 with CUDA capability sm_75 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
</code></pre>
<p>How come the pre-built PyTorch binaries for 1.6 do not support the newest CUDA capability? Is this intentional? How can I fix my setup? Thanks!</p>

kl_divergence: <p>Even I’m facing this issue with <code>Quadro RTX 8000</code>. The official recommendation is to build from source. But I don’t get it, why is not supported ?</p>

ptrblck: <p><a class="mention" href="/u/seemethere">@seemethere</a> was <code>sm_75</code> dropped accidentally, since the release notes only mention <code>6.1</code> and <code>PTX for 3.7</code>?<br>
However, <a href="https://github.com/pytorch/builder/commit/8d064bfcfe2e73d89a067dd5f9311828c1792f90">this PR</a> seems to have also removed <code>7.5</code>.</p>

ilovescience: <p>I also have the same warning on Colab Tesla T4 GPU. But it doesn’t seem to actually affect anything I can still use CUDA.</p>

kl_divergence: <p>I’m able to use but not sure whether all CUDA functionalities work as intended because of this error. This is a strange behavior. Building Pytorch isn’t easy thing on my restricted server.</p>

tom: <p>I think it’s there for cuda 10.2. (But my poor 1080 Ti is dropped.)</p>

bsun0802: <p>Facing the same issue here. Card is GeForce RTX 2080 Ti. What does this warning mean? Inside torch, torch.cuda.is_available() returns True and I can train with GPU. Do I need to and how can I fix it?</p>
<p>Thanks in advance.</p>
<pre><code class="lang-auto">/home/zbxs/miniconda3/lib/python3.7/site-packages/torch/cuda/__init__.py:125: UserWarning:
GeForce RTX 2080 Ti with CUDA capability sm_75 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the GeForce RTX 2080 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
</code></pre>
<p>Python=3.7.7<br>
cudatoolkit               10.1.243             h6bb024c_0    defaults<br>
Pytorch 1.6.0.</p>

ptrblck: <p>You are right. I missed the <code>7.5</code> argument for CUDA10.2</p>
<p><a class="mention" href="/u/bsun0802">@bsun0802</a> as <a class="mention" href="/u/tom">@tom</a> mentioned, <code>sm_75</code> is still available in the CUDA10.2 binary, so you might want to install this particular version.</p>

sniklaus: <p>What is the rationale behind removing <code>sm_75</code> for the binaries that target CUDA 10.1? I am afraid that I cannot upgrade to CUDA 10.2 due to dependency and driver issues.</p>
<p>What are the effects of using the PyTorch binaries that do not support <code>sm_75</code> on a graphics card that is supporting <code>sm_75</code>?</p>

smth: <p>tagging <a class="mention" href="/u/seemethere">@seemethere</a>, we need to fix this bug and re-push the CUDA 10.1 binaries.</p>

smth: <p>okay, I got an update. <a class="mention" href="/u/seemethere">@seemethere</a> is fixing the issue and re-uploading the cuda 10.1 binaries asap, like in ~3 hours or so</p>

sniklaus: <p>Awesome, thank you for taking care of this so quickly!</p>

sniklaus: <p>Just a heads-up for everyone following along, the issue has been <a href="https://github.com/pytorch/builder/pull/472" rel="nofollow noopener">fixed</a> underneath the hood but the binaries have unfortunately not been update yet: <a href="https://anaconda.org/pytorch/pytorch/files" rel="nofollow noopener">https://anaconda.org/pytorch/pytorch/files</a></p>

sniklaus: <p>The new binaries seem to be available now, at least for <code>pip</code>. Make sure to use <code>--no-cache-dir</code> to not use a locally cached binary when <code>pip</code> installing PyTorch again.</p>
<pre><code class="lang-auto">pip install --no-cache-dir torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>

smth: <p>conda binaries shouldn’t have been broken, only the wheels were broken, and they are fixed now. thanks for patience <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

sniklaus: <p>Thank you for chiming in! I can confirm, the new binaries showed up on pip a few hours or so ago. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>

pesenti: <p>My code (on Colab Tesla T4 GPU) seems to work despite the warning. What is the impact of this bug?</p>

ptrblck: <p>The <code>sm_70</code> argument <em>should</em> add the sass and ptx to the binaries. The ptx would make the code executable for your T4 (compute capability 7.5), but might not yield the optimal performance.</p>

malfet: <p>To the best of my knowledge, there are no sm_70 ptx in pytorch-1.6 binaries, but sm_70 ISA is compatible with sm_75, although some kernels might not launch, if I’m understating this correctly (from <a href="https://docs.nvidia.com/cuda/turing-compatibility-guide/index.html" class="inline-onebox" rel="noopener nofollow ugc">1. Turing Compatibility — Turing Compatibility Guide 12.3 documentation</a> ):</p>
<blockquote>
<p>The Turing architecture is based on Volta’s Instruction Set Architecture ISA 7.0, extending it with new instructions. As a consequence, any binary that runs on Volta will be able to run on Turing (forward compatibility), but a Turing binary will not be able to run on Volta. Please note that Volta kernels using more than 64KB of shared memory (via the explicit opt-in, see CUDA C++ Programming Guide) will not be able to launch on Turing, as they would exceed Turing’s shared memory capacity.</p>
</blockquote>

luiszeni: <p>This removed the warnings for me. Tks</p>