[Question] I have a question about installing PyTorch

e1920019: <p>Hello</p>
<p>I have a question for experts. I’m trying to install PyTorch, but I’m encountering errors. I have installed CUDA 12.3.0. Is PyTorch not compatible with CUDA 12.3.0? Please help.</p>

ptrblck: <p>PyTorch is compatible with CUDA 12.3 and will use your locally installed CUDA toolkit for source builds. The binaries ship with their own CUDA dependencies, won’t use your local CUDA toolkit, and only a properly installed NVIDIA driver is needed.</p>

e1920019: <p>hank you for your response.</p>
<p>Despite upgrading to CUDA 12.3.0, installing the appropriate NVIDIA driver for your PC, and updating pip to the latest version, you are still encountering an error. The error message is as follows:</p>
<p>“Looking in links: <a href="https://download.pytorch.org/whl/cu123/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu123/torch_stable.html</a><br>
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)<br>
ERROR: No matching distribution found for torch”</p>
<p>The command you used is: pip install torch torchvision torchaudio -f <a href="https://download.pytorch.org/whl/cu123/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu123/torch_stable.html</a></p>
<p>To address this issue, please ensure that the CUDA version you have (CUDA 12.3.0) is officially supported by checking the PyTorch website. Additionally, double-check the version specifier for <code>torch</code> in your installation command to ensure it is correct. Modify your command accordingly,</p>
<p>Make sure to adjust the CUDA version in the command to match your setup. If the issue persists, consider trying different combinations or checking the PyTorch GitHub repository for any updates or changes in supported versions.</p>
<p>Feel free to provide additional details or logs if the problem persists, and I’ll do my best to assist you.</p>

ptrblck: <p>I don’t know if you’ve just posted a response from a bot, but <code>https://download.pytorch.org/whl/cu123/torch_stable.html</code> is an invalid URL and you should use the provided ones from <a href="https://pytorch.org/get-started/locally/">here</a>.</p>

e1920019: <p>The issue has been resolved. Thank you for your guidance.</p>

barista: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="191829" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>PyTorch is compatible with CUDA 12.3 and will use your locally installed CUDA toolkit for source builds. The binaries ship with their own CUDA dependencies, won’t use your local CUDA toolkit, and only a properly installed NVIDIA driver is needed.</p>
</blockquote>
</aside>
<p>So, PyTorch 2.1.1 (and the nightly builds) can effectively run with CUDA version 12.3, and there is no need to downgrade to 12.1? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <aside class="quote no-group" data-username="barista" data-post="6" data-topic="191829">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/barista/48/47476_2.png" class="avatar"> barista:</div>
<blockquote>
<p>So, PyTorch 2.1.1 (and the nightly builds) can effectively run with CUDA version 12.3</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="191829">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The binaries ship with their own CUDA dependencies, won’t use your local CUDA toolkit,</p>
</blockquote>
</aside>
<p>No, you would need to build from source if you need CUDA 12.3 as explained before.</p>

SuperSonnix71: <p>I have the same issue. where  llama fails running on the GPU.<br>
Traced it to torch!</p>
<hr>
<h2><a name="nvidia-smi-5452906-driver-version-5452906-cuda-version-123-1" class="anchor" href="#nvidia-smi-5452906-driver-version-5452906-cuda-version-123-1"></a>| NVIDIA-SMI 545.29.06  | Driver Version: 545.29.06  | CUDA Version: 12.3  |</h2>
<h2><a name="nvcc-nvidia-r-cuda-compiler-driver-copyright-c-2005-2023-nvidia-corporation-built-on-fri_nov__3_171649_pdt_2023-cuda-compilation-tools-release-123-v123103-build-cuda_123r123compiler33492891_0-2" class="anchor" href="#nvcc-nvidia-r-cuda-compiler-driver-copyright-c-2005-2023-nvidia-corporation-built-on-fri_nov__3_171649_pdt_2023-cuda-compilation-tools-release-123-v123103-build-cuda_123r123compiler33492891_0-2"></a>nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2023 NVIDIA Corporation<br>
Built on Fri_Nov__3_17:16:49_PDT_2023<br>
Cuda compilation tools, release 12.3, V12.3.103<br>
Build cuda_12.3.r12.3/compiler.33492891_0</h2>
<h2><a name="export-pathusrlocalcuda-123binpath-export-ld_library_pathusrlocalcuda-123lib64ld_library_path-3" class="anchor" href="#export-pathusrlocalcuda-123binpath-export-ld_library_pathusrlocalcuda-123lib64ld_library_path-3"></a>export PATH=/usr/local/cuda-12.3/bin:$PATH<br>
export LD_LIBRARY_PATH=/usr/local/cuda-12.3/lib64:$LD_LIBRARY_PATH</h2>
<p>python -c “import torch; print(‘PyTorch Version:’, torch.<strong>version</strong>, ‘\nCUDA Version:’, torch.version.cuda, ‘\nCUDA Available:’, torch.cuda.is_available())”</p>
<h2><a name="pytorch-version-211-cuda-version-121-cuda-available-false-4" class="anchor" href="#pytorch-version-211-cuda-version-121-cuda-available-false-4"></a>PyTorch Version: 2.1.1<br>
CUDA Version: 12.1<br>
CUDA Available: False</h2>
<p>llama_new_context_with_model: compute buffer total size = 278.43 MiB<br>
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |<br>
16:54:47.668 [INFO    ] private_gpt.components.embedding.embedding_component - Initializing the embedding model in mode=local<br>
16:54:48.561 [WARNING ]               py.warnings - /home/anaconda3/envs/privategpt/lib/python3.11/site-packages/torch/cuda/<strong>init</strong>.py:611: UserWarning: Can’t initialize NVML<br>
warnings.warn(“Can’t initialize NVML”)</p>

3co: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your clarification.  I’ll proceed with a manual build of Torch with CUDA 12.3 now…</p>

Ciberponk: <p>Refer to <a href="https://github.com/pytorch/pytorch/issues/112500" class="inline-onebox" rel="noopener nofollow ugc">Bug Report No Version Of Pytorch for cuda 12.3 · Issue #112500 · pytorch/pytorch · GitHub</a><br>
with Python 3.12.1 and CUDA 12.3, the Preview (Nightly) version of PyTorch works <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
OS: Windows 10 64bit</p>

jmvd: <p>Hello,<br>
I’m quite new in these endeavors, can you post a full example how to build from source with CUDA 12.3 in a Conda environment?<br>
Thank very much.</p>