Not able to include cusolverDn.h

PyTorch94: <p>I installed Pytorch in Ubuntu 20.0 using both conda command and pip3 command (in a conda environment).<br>
However, I am not able to find the cusolverDn.h header anywhere inside the anaconda3 folder.<br>
I am using ATen/cuda/CUDAContext.h header file in CUDA implementation, which depends on cusolverDn.h (as the error message says).<br>
Please help in rectifying the issue.</p>

ptrblck: <p>The cuSOLVER headers would ship with your locally installed CUDA toolkit. Could you explain your use case a bit more as it seems you are trying to build PyTorch from source using the conda binaries?</p>

PyTorch94: <p>I was trying to work with the official deformable-DETR network, which has the deformable attention module as a CUDA implementation. While implementing it, they included the torch library header files for ATen and others, including the one mentioned in the query.<br>
I am working on a system with Ubuntu 20.04 and anaconda as a package manager (conda 22.11.1). The nvcc version is v10.1.243. I tried installing PyTorch using both the conda command and the pip3 given in the official website. I didn’t install it from the source. But in both cases, VS Code Intellisense threw an error that cusolverDn.h can not be opened. upon searching for the file, I could not find it.</p>

ptrblck: <p>The PyTorch binaries do not ship with the entire CUDA toolkit and all of its headers.<br>
If you want to build Deformable-DETR from source, make sure to install PyTorch as one dependency, but use  your locally installed CUDA toolkit (including the cuSOLVER headers) to build the DETR lib.</p>

HeyangQin: <p>I think this may be related to the recent change in the conda packages. I just created a new conda environment, installed pytorch according to the official documentation(<code>conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia</code>) and tried to compile apex from source. The <code>pip</code> commandline looks like this:</p>
<pre><code class="lang-auto">/vc_data/users/heyangqin/anaconda3/envs/deepspeed/bin/nvcc  -I/vc_data/users/heyangqin/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include -I/vc_data/users/heyangqin/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -I/vc_data/users/heyangqin/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/TH -I/vc_data/users/heyangqin/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/THC -I/vc_data/users/heyangqin/anaconda3/envs/deepspeed/include -I/vc_data/users/heyangqin/anaconda3/envs/deepspeed/include/python3.10 -c -c /vc_data/users/heyangqin/apex/csrc/multi_tensor_sgd_kernel.cu -o /vc_data/users/heyangqin/apex/build/temp.linux-x86_64-cpython-310/csrc/multi_tensor_sgd_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 --threads 4 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_70,code=compute_70 -gencode=arch=compute_70,code=sm_70 -std=c++14
</code></pre>
<p>This <code>pip</code> compile commandline calls the <code>nvcc</code> in the conda env and it does not include the system CUDA dir <code>/usr/local/cuda/include/</code> where the <code>cusolverDn.h</code> locates which causes the error. So I manually updated the <code>PATH</code> by <code>export PATH=/usr/local/cuda/bin:$PATH</code> and the error is gone. I wonder if this is the intended behavior?</p>

ptrblck: <p>Your explanation sounds right and I don’t think <code>nvcc</code> is supposed to ship as a dependency and be usable for source build from the <code>conda</code> environment (at least we are not testing this use case).<br>
Using the local CUDA toolkit is the right approach and we should try to filter out some packages for the next release.</p>

senthilps8: <p>I ran into the same issue because <code>nvcc</code> has been packaged with <code>pytorch</code> in the latest release. In my case, the fix was not a trivial as updating the <code>PATH</code> variable. A package compiled with the local <code>nvcc</code> throws this error: <code>provided PTX was compiled with an unsupported toolchain</code>. From this <a href="https://forums.developer.nvidia.com/t/provided-ptx-was-compiled-with-an-unsupported-toolchain-error-using-cub/168292/1" rel="noopener nofollow ugc">link</a>, it looks like the issue might be that driver version expected by the pytorch nvcc is different from the local nvcc. Unfortunately, I cannot update my driver and I still cannot figure out a real solution but I have a temporary workaround for my use case. Just leaving it here in case others run into this. Hopefully the next releases will revert back to not including <code>nvcc</code>.</p>

hamedB: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I am facing with such cusolverDn.h issue. The reason is that I am trying to install the deformable convolution from this <a href="https://github.com/jdonnelly36/Deformable-ProtoPNet" rel="noopener nofollow ugc">link</a>. It has a setup file in the folder " Deformable-Convolution-V2-PyTorch".<br>
The exact error I face is as follow.</p>
<blockquote>
<p>"LOCALPATH/anaconda3/envs/pytorch_1_13/lib/python3.9/site-packages/torch/include/ATen/cuda/CUDAContext.h:10:10: fatal error: cusolverDn.h: No such file or directory</p>
<p>10 | <span class="hashtag-raw">#include</span> &lt;cusolverDn.h&gt;<br>
|          ^~~~~~~~~~~~~~<br>
compilation terminated.<br>
error: command ‘/usr/bin/gcc’ failed with exit code 1<br>
"</p>
</blockquote>
<p>As error shows cusolverDN.h does not exit. However, the file CUDAContext.h does exist.<br>
I have upgraded my nvidia cuda version to 12.2. My pytorch version is 1.13.1 and the pytorch-cuda is 11.6.</p>
<p>When I want to run the setup.py file from terminal, I activate my anaconda env. So, it uses locally installed CUDA toolkit. Can you explain more what other things I can try to fix this error?</p>
<p>Thanks a lot for your attention.</p>

ptrblck: <p>You could search for this file on your system, which should point to a the locally installed CUDA toolkit and add the path into the includes if needed (I would assume the 3rd party repository already does this).<br>
E.g. on my system I see:</p>
<pre><code class="lang-auto">find /usr/ -name cusolverDn.h
/usr/local/cuda-12.3/targets/x86_64-linux/include/cusolverDn.h
</code></pre>

hamedB: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your quick responds.<br>
I can not find cusolverDn.h in my system.<br>
I just installed anaconda 3, followed by installing an env. including pytorch libraries.<br>
Would you please tell me what other package(s) I should install to have cusolverDn.h?</p>
<p>Thanks again.</p>

ptrblck: <p>You should install a full CUDA toolkit from: <a href="https://developer.nvidia.com/cuda-downloads">https://developer.nvidia.com/cuda-downloads</a></p>

hamedB: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a> . I have just installed the Base Installer package from the link and the cusolverDn.h was added to the following address in my machine.</p>
<blockquote>
<p>/usr/local/cuda-12.3/targets/x86_64-linux/include/cusolverDn.h</p>
</blockquote>
<p>I didn’t install again the NVIDIA Driver as I have it.<br>
A silly question <img src="https://discuss.pytorch.org/images/emoji/apple/see_no_evil.png?v=12" title=":see_no_evil:" class="emoji" alt=":see_no_evil:" loading="lazy" width="20" height="20">, The part related to adding the path to the includes is not clear for me .  would you explain where I should add it?</p>

hamedB: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  I added the path including this .h file to $PATH. However, I still receive the same error before. I am not sure if I got what you mean about adding into includes.</p>

hamedB: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I was able to solve the issue with the file cusolverDn.h by finding it and adding its path to $PATH. Now the installation process goes further and faces a new error as follow.</p>
<blockquote>
<p>home/hamed/PycharmProjects/Deformable_ProtoPNet/Deformable-Convolution-V2-PyTorch/src/cuda/deform_psroi_pooling_cuda.cu:18:10: fatal error: THC/THC.h: No such file or directory<br>
18 | <span class="hashtag-raw">#include</span> &lt;THC/THC.h&gt;<br>
|          ^~~~~~~~~~~<br>
compilation terminated.<br>
error: command ‘/usr/local/cuda/bin/nvcc’ failed with exit code 1</p>
</blockquote>
<p>Has THC/THC been removed from Pytorch 2.1.2?</p>

ptrblck: <p>Yes, the <code>THC</code> namespace is dead as described in <a href="https://discuss.pytorch.org/t/question-about-thc-thc-h/147145/3">this topic</a> so you would need to update your code to use the new <code>ATen</code> API.</p>

ssbarve2: <p>I was facing this similar issue. In my case specifically, compiler wasn’t looking for <code>cusolverDn.h</code> at the right location.<br>
Where it is present:</p>
<pre><code class="lang-auto">$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
V11.6
$ find /usr/ -name cusolverDn.h
/usr/local/cuda-11.6/targets/x86_64-linux/include/cusolverDn.h
</code></pre>
<p>ERROR: Compiler was looking for it within the pytorch package.</p>
<pre><code class="lang-auto">/../miniconda3/envs/nerfacc/lib/python3.9/site-packages/torch/include/ATen/cuda/CUDAContext.h:10:10: fatal error: cusolverDn.h: No such file or directory
   10 | #include &lt;cusolverDn.h&gt;
</code></pre>
<p>Added the following to my bashrc script, resolved the issue. (adding path to cuda toolkit in $PATH var didn’t help in my case)</p>
<pre><code class="lang-auto">export CPLUS_INCLUDE_PATH=/usr/local/cuda-11.6/include:$CPLUS_INCLUDE_PATH
export C_INCLUDE_PATH=/usr/local/cuda-11.6/include:$C_INCLUDE_PATH
</code></pre>