PyTorch 2.1 compatibility with CUDA 12.3

SuperSonnix71: <p>llama fails running on the GPU.<br>
Traced it to torch! Torch is using CUDA 12.1.Tried multiple different approaches where I removed 12.1 to make it use 12.3 downgraded the Nvidia driver. No joy! All help is appreciated. Running on a openSUSE tumbleweed.<br>
PyTorch Version: 2.1.1<br>
CUDA Version: 12.1<br>
CUDA Available: False</p>
<hr>
<h2><a name="nvidia-smi-5452906-driver-version-5452906-cuda-version-123-1" class="anchor" href="#nvidia-smi-5452906-driver-version-5452906-cuda-version-123-1"></a>| NVIDIA-SMI 545.29.06 | Driver Version: 545.29.06 | CUDA Version: 12.3 |</h2>
<h2><a name="nvcc-nvidia-r-cuda-compiler-driver-2" class="anchor" href="#nvcc-nvidia-r-cuda-compiler-driver-2"></a>nvcc: NVIDIA (R) Cuda compiler driver</h2>
<p>Copyright (c) 2005-2023 NVIDIA Corporation<br>
Built on Fri_Nov__3_17:16:49_PDT_2023<br>
Cuda compilation tools, release 12.3, V12.3.103<br>
Build cuda_12.3.r12.3/compiler.33492891_0</p>
<h2><a name="export-pathusrlocalcuda-123binpath-3" class="anchor" href="#export-pathusrlocalcuda-123binpath-3"></a>export PATH=/usr/local/cuda-12.3/bin:$PATH</h2>
<p>export LD_LIBRARY_PATH=/usr/local/cuda-12.3/lib64:$LD_LIBRARY_PATH</p>
<p>python -c “import torch; print(‘PyTorch Version:’, torch.<strong>version</strong>, ‘\nCUDA Version:’, torch.version.cuda, ‘\nCUDA Available:’, torch.cuda.is_available())”</p>
<h2><a name="pytorch-version-211-4" class="anchor" href="#pytorch-version-211-4"></a>PyTorch Version: 2.1.1</h2>
<p>CUDA Version: 12.1<br>
CUDA Available: False</p>
<p>llama_new_context_with_model: compute buffer total size = 278.43 MiB<br>
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |<br>
16:54:47.668 [INFO ] private_gpt.components.embedding.embedding_component - Initializing the embedding model in mode=local<br>
16:54:48.561 [WARNING ] py.warnings - /home/anaconda3/envs/privategpt/lib/python3.11/site-packages/torch/cuda/<strong>init</strong>.py:611: UserWarning: Can’t initialize NVML<br>
warnings.warn(“Can’t initialize NVML”)</p>

ptrblck: <p>Check your NVIDIA driver and reinstall it if needed.</p>

SuperSonnix71: <p>Already did! tried with the following versions. No joy!<br>
NVIDIA-Linux-x86_64-535.129.03.run<br>
NVIDIA-Linux-x86_64-545.29.06.run</p>

SuperSonnix71: <p>The issue is now solved!</p>

ptrblck: <p>Could you post your solution here?</p>

banton: <p>My friend, if you are prepared to receive help on a technical forum, please also be prepared to post the solution if you arrive to it yourself.</p>

Aymen_ABID: <p>have you an idea to install it for.</p>
<p>nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2023 NVIDIA Corporation<br>
Cuda compilation tools, release 12.3, V12.3.107<br>
Build cuda_12.3.r12.3/compiler.33567101_0</p>
<p>to use resolve:<br>
import torch<br>
print(torch.cuda.is_available())<br>
=&gt;False</p>

ptrblck: <p>Follow <a href="https://github.com/pytorch/pytorch#from-source">these instructions</a> and it should work.</p>

Aymen_ABID: <blockquote>
<p>pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu121" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu121</a><br>
Defaulting to user installation because normal site-packages is not writeable<br>
Looking in indexes: <a href="https://download.pytorch.org/whl/cu121" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu121</a><br>
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)<br>
ERROR: No matching distribution found for torch</p>
</blockquote>
<p>Trying</p>
<pre><code class="lang-auto">pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 --no-cache-dir
</code></pre>
<p>?</p>

ptrblck: <p>You asked your initial question in this topic which is specific to CUDA 12.3. If you really need this version for some reason you would need to build PyTorch from source as indicated in my previous answer.<br>
Now you post a <code>pip install</code> command instead, which would install PyTorch binaries with CUDA dependencies.<br>
Could you explain what you actually want?</p>

Aymen_ABID: <aside class="quote no-group quote-modified" data-username="Aymen_ABID" data-post="7" data-topic="192659">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aymen_abid/48/67194_2.png" class="avatar"> Aymen_ABID:</div>
<blockquote>
<ol start="12">
<li></li>
</ol>
</blockquote>
</aside>
<p>I want to use the GPU NVIDIA via jupyter…</p>
<p>after installing<br>
with (!pip install torch torchvision torchaudio -f <a href="https://download.pytorch.org/whl/cu123/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu123/torch_stable.html</a>)<br>
or<br>
(!pip3 install torch torchvision torchaudio -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a>)</p>
<p>it seem good, only if it is not for :</p>
<blockquote>
<p>**Defaulting to user installation because normal site-packages is not writeable …<br>
Requirement already satisfied…</p>
</blockquote>
<p><strong>But</strong><br>
Unfortunately, the response is the same.</p>
<div><br>import torch torch.cuda.is_available()</div>
<p>import torch</p>
<p>torch.cuda.is_available()</p>
<p>Out[1]:</p>
<p>False</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji only-emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Uninstall the current PyTorch binary and just run a supported commend from the <a href="https://pytorch.org/get-started/locally/">install matrix</a>.</p>

Aymen_ABID: <p>(base) C:\Users\aymen&gt;conda uninstall pytorch<br>
Collecting package metadata (repodata.json): done<br>
Solving environment: failed</p>
<p>PackagesNotFoundError: The following packages are missing from the target environment:</p>
<ul>
<li>pytorch</li>
</ul>

ptrblck: <p>How are you importing <code>torch</code> if nothing was installed?<br>
In the end you should either uninstall all PyTorch binaries and install the current stable or nightly release (a simple <code>pip install torch</code> would be enough to install <code>torch==2.1.2+cu121</code>) or create a new and empty virtual environment and install PyTorch there.</p>

Aymen_ABID: <p>I have eraze all (conda )<br>
install again same prbs<br>
actually I have this one</p>
<blockquote>
<p>The following packages will be UPDATED:</p>
<p>pytorch            pkgs/main::pytorch-2.1.0-cpu_py311hd0~ → pytorch::pytorch-2.1.2-py3.11_cuda12.1_cudnn8_0<br>
torchvision        pkgs/main::torchvision-0.15.2-cpu_py3~ → pytorch::torchvision-0.16.2-py311_cu121</p>
<p>Proceed ([y]/n)? y</p>
<p>Downloading and Extracting Packages</p>
<p>CancelledError()<br>
CancelledError()<br>
CancelledError()<br>
CondaError: Downloaded bytes did not match Content-Length<br>
url: <a href="https://conda.anaconda.org/pytorch/win-64/pytorch-2.1.2-py3.11_cuda12.1_cudnn8_0.tar.bz2" rel="noopener nofollow ugc">https://conda.anaconda.org/pytorch/win-64/pytorch-2.1.2-py3.11_cuda12.1_cudnn8_0.tar.bz2</a><br>
target_path: C:\ProgramData\anaconda3\pkgs\pytorch-2.1.2-py3.11_cuda12.1_cudnn8_0.tar.bz2<br>
Content-Length: 1339081254<br>
downloaded bytes: 53705125</p>
<p>CancelledError()<br>
CancelledError()</p>
</blockquote>
<p>for<br>
(base) C:\Users\aymen&gt;conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia</p>
<p>but</p>
<blockquote>
<p>The following packages will be UPDATED:</p>
<p>pytorch            pkgs/main::pytorch-2.1.0-cpu_py311hd0~ → pytorch::pytorch-2.1.2-py3.11_cuda11.8_cudnn8_0<br>
torchvision        pkgs/main::torchvision-0.15.2-cpu_py3~ → pytorch::torchvision-0.16.2-py311_cu118</p>
<p>Proceed ([y]/n)? y</p>
<p>Downloading and Extracting Packages</p>
<p>Preparing transaction: done<br>
Executing transaction: done</p>
</blockquote>
<p>is work!</p>
<p>Else</p>
<blockquote>
<p>torch.cuda.is_available()</p>
</blockquote>
<p>give FALSE unfortunately!</p>

jesmine: <p>brother, have you got the solution ? my cuda version is 12.3 . I’ve installed cuda , cudnn but cant install pytoch.  "import  torch<br>
print (torch.<strong>version</strong>)<br>
print (torch.cuda.is_available())<br>
2.3.0+cpu<br>
False " please let us know</p>

jesmine: <p>my cuda version is 12.3 . I’ve installed cuda , cudnn but cant install pytoch. "import torch<br>
print (torch.<strong>version</strong> )<br>
print (torch.cuda.is_available())<br>
2.3.0+cpu<br>
False " please let us know</p>

ptrblck: <p>You’ve installed the CPU-only binary and would need to install the PyTorch binary with CUDA runtime dependencies by selecting it in the install matrix on our webite.</p>

jesmine: <p>thank you for reply. I’ve tried several times to install " stable = windows= conda= python= cuda (12.1) " but the installation was never completed.  it says "(‘Connection broken: IncompleteRead(23132496 bytes read, 1201416417 more expected)’, IncompleteRead(23132496 bytes read) &amp; Downloading and Extracting Packages:</p>
<p>CondaHTTPError: HTTP 000 CONNECTION FAILED for url <a href="https://conda.anaconda.org/nvidia/win-64/libcusolver-dev-11.4.4.55-" rel="noopener nofollow ugc">https://conda.anaconda.org/nvidia/win-64/libcusolver-dev-11.4.4.55-</a><br>
Elapsed: -</p>
<p>An HTTP error occurred when trying to retrieve this URL.<br>
HTTP errors are often intermittent, and a simple retry will get you on your way.</p>
<p>CondaHTTPError: HTTP 000 CONNECTION FAILED for url <a href="https://conda.anaconda.org/nvidia/win-64/libcusolver-dev-11.4.4.55-" rel="noopener nofollow ugc">https://conda.anaconda.org/nvidia/win-64/libcusolver-dev-11.4.4.55-</a><br>
Elapsed: -</p>
<p>An HTTP error occurred when trying to retrieve this URL.<br>
HTTP errors are often intermittent, and a simple retry will get you on your way.</p>
<p>(base) PS C:\Users\hp&gt;"  and one more thing – when i checked "conda list " it has the “torch     2.3.0+cu121     pypi_0    pypi &amp; torchaudio   2.3.0+cu121              pypi_0    pypi” but can’t find any pytorch version neither cpu nor cuda.   tried both ways “conda install” and “pip3 install”</p>
<p>" import torch<br>
print(torch.<strong>version</strong>)<br>
print(torch.version.cuda)<br>
print(torch.randn(1).cuda())</p>
<p>2.3.0+cpu<br>
None<br>
AssertionError: Torch not compiled with CUDA enabled</p>
<p>what should i do now?please let me know what to do next</p>

ptrblck: <p>The error points to a connection issue when trying to download packages from <code>conda</code>, so you might want to check your connection or try to download it later as suggested in the error message.</p>