Lowest version of PyTorch supporting CUDA 11.8?

davidw82: <p>Hi,<br>
What is the lowest version number of PyTorch that started supporting CUDA 11.8? Do I need to go all the way up to PyTorch 1.13 to get support?<br>
And to be clear, I’m always compiling from source.</p>
<p>Cheers,<br>
David</p>

ptrblck: <p>I don’t see any CUDA 11.8 - specific changes to enable source builds (besides enabling the Hopper architecture, if needed) and we were using a <code>1.13.0</code>-prerelease in our first 11.8 container as seen <a href="https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-22-09.html#rel-22-09">here</a>.</p>

davidw82: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> , and also thanks for providing the link to the PyTorch containers.</p>

nicolaspanel: <p>thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for the link <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"><br>
It seems that upgrading to cuda 11.8 <a href="https://discuss.pytorch.org/t/ddp-training-on-rtx-4090-ada-cu118/168366">breaks DDP</a> (at least for ADA arch or RTX 4090 serie)<br>
Are you aware of this issue ?<br>
Do you know a workaround other than training on a single GPU ?<br>
Best</p>

ptrblck: <p>I doubt CUDA 11.8 breaks DDP workloads as I haven’t seen any failures so far with it.<br>
Check a few similar issues posted in this discussion board and post the missing information about your workflow, environment, as well as the log outputs using the debug flags.</p>

nicolaspanel: <p>thanks <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
After further investigation the problem was due to NCCL backend trying to use peer to peer (P2P) transport.<br>
Forcing <code>NCCL_P2P_DISABLE=1</code> fixed the issue <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>

akshu2023: <p>Seems to be a known issue. Unsure who is addressing this between nvidia and amd.</p>
<aside class="quote" data-post="12" data-topic="168366">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7feea3/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/ddp-training-on-rtx-4090-ada-cu118/168366/12">DDP training on RTX 4090 (ADA, cu118)</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    Seems to be AMD specific issue with multiple 4090s NCCL P2P functionality. Unsure who will resolve it when.
  </blockquote>
</aside>


milenna89: <p>How can I install torch 1.13.1 or anything lower than torch 2.0 when I have cuda 11.8 if I want to do that without installing from source with one of the commands <a href="https://pytorch.org/get-started/previous-versions/" rel="noopener nofollow ugc">here</a>? there’s only one example for cuda 11.8 but it is given for torch 2.0<br>
when I run</p>
<pre><code class="lang-auto">conda install pytorch==1.13.1  pytorch-cuda=11.8 -c pytorch -c nvidia
</code></pre>
<p>Torch ends up being installed without cuda support since <code>torch.version.cuda</code> is empty and<br>
<code>torch.zeros(1).cuda()</code> gives</p>
<pre><code class="lang-auto">
&lt;stdin&gt;:1: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /opt/conda/conda-bld/pytorch_1670525493953/work/torch/csrc/utils/tensor_numpy.cpp:77.)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File ".../python3.10/site-packages/torch/cuda/__init__.py", line 221, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
</code></pre>

ptrblck: <p>The PyTorch 1.13.1 binaries were built with 11.6 and 11.7 as given <a href="https://pytorch.org/get-started/previous-versions/#linux-and-windows-2">here</a>.<br>
Note that the binaries ship with their own CUDA dependencies and your locally installed CUDA toolkit will be used if you build PyTorch from source or a custom CUDA extension.<br>
In case you need to run <code>torch==1.13.1</code> with CUDA 11.8 you would thus have to build it from source.</p>

aditya28: <p>hello, i am having a similar problem<br>
I’m a complete beginner<br>
I’m using tensorflow-gpu 2.10, CUDA 11.8, pytorch 1.13.1 and python 3.8.8<br>
somehow tensorflow recognizes my CUDA &amp; gpu but pytorch doesn’t<br>
tf.test.is_gpu_available() &amp; tf.test.is_built_with_cuda() both return True<br>
but<br>
torch.cuda.is_available() returns false</p>
<p>i am trying to install pytorch using<br>
pip3 install torch==1.13.1+cu118 torchvision torchaudio<br>
but gives errors<br>
ERROR: Could not find a version that satisfies the requirement torch==1.13.1+cu118 (from versions: 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2)<br>
ERROR: No matching distribution found for torch==1.13.1+cu118</p>
<p>I have reinstalled tensorflow-gpu and torch after CUDA installation</p>
<p>please help and thanks in advance</p>