Couldn't find CUDA library root

username: olagarde
cooked: <p>Need help finding what’s actually causing the cmake failure;  build fails wth this msg despite finding the CUDA root and correctly populating the cmake cache with the root and toolkit_root and associated libs.  CMake error log ends with failed ninja tests for alternatives immediately prior to this (testing avx512f support for fbgemm build), no indication there or in output log as to what’s actually failing.  This is not a repeat of <span class="hashtag">#23066</span> or similar, the cmake runtime is actually locating the cuda/cudart/cudnn/etc root/toolkitroot/etc correctly per the ccache.</p>
<p>CUDA 11.4.2-1 el7 x86_6 from NVida repo with all the usual -nn, -rt, -nr, parser, solver, blas, lapack, etc, gubbins installed.  GCC 11.2.0 and python 3.10.0 built from source in /opt/soft/, resulting pip3 used to install all prereqs in /opt/soft[/lib/python].  Suggestion to move cuda to raw build in /usr/local is not permitted, cuda must be installed via nvidia repo packages keysigned gpg-pubkey-f90c0e97-483e8383 and installing into /usr/include/cuda and /lib[64].</p>

username: olagarde
cooked: <p>Build procedure is:</p>
<p>umask 022<br>
export PATH="/opt/soft/bin:$PATH"<br>
export PYTHONPATH=/opt/soft/lib/python:$PYTHONPATH<br>
export LD_LIBRARY_PATH="/opt/soft/libexec/gcc/x86_64-pc-linux-gnu/11.2.0:$LD_LIBRARY_PATH"<br>
export LD_LIBRARY_PATH="/opt/soft/lib/gcc/x86_64-pc-linux-gnu/11.2.0/plugin:$LD_LIBRARY_PATH"<br>
export LD_LIBRARY_PATH="/opt/soft/lib/gcc/x86_64-pc-linux-gnu/11.2.0:$LD_LIBRARY_PATH"<br>
export LD_LIBRARY_PATH="/opt/soft/lib64:$LD_LIBRARY_PATH"<br>
git clone --recursive <a href="https://github.com/pytorch/pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a><br>
cd pytorch<br>
export CMAKE_PREFIX_PATH=/opt/soft<br>
export ROCM=0<br>
export CUDA_HOME=/<br>
python3 ./setup.py build</p>
<p>StdOut:<br>
/zvs0qfDB</p>
<p>Cmake cache:<br>
/r5qpsmbf</p>
<p>Output log:<br>
/z8MRQiLN</p>
<p>Error log:<br>
/upKn6u88</p>

username: olagarde
cooked: <p>Yes those are pastebins, “new users can only…”, etc.</p>

username: olagarde
cooked: <p>Tried a few other things, found this in CMakeDetermineCUDACompiler.cmake:</p>
<pre><code class="lang-auto">                    # CMAKE_CUDA_COMPILER_LIBRARY_ROOT contains the device library.
                    if(EXISTS "${CMAKE_CUDA_COMPILER_TOOLKIT_ROOT}/nvvm/libdevice")
                      set(CMAKE_CUDA_COMPILER_LIBRARY_ROOT "${CMAKE_CUDA_COMPILER_TOOLKIT_ROOT}")
                    elseif(CMAKE_SYSROOT_LINK AND EXISTS "${CMAKE_SYSROOT_LINK}/usr/lib/cuda/nvvm/libdevice")
                      set(CMAKE_CUDA_COMPILER_LIBRARY_ROOT "${CMAKE_SYSROOT_LINK}/usr/lib/cuda")
                    elseif(EXISTS "${CMAKE_SYSROOT}/usr/lib/cuda/nvvm/libdevice")
                      set(CMAKE_CUDA_COMPILER_LIBRARY_ROOT "${CMAKE_SYSROOT}/usr/lib/cuda")
                    else()
                      message(FATAL_ERROR "Couldn't find CUDA library root.")
                    endif()

</code></pre>
<p>So evidently it’s static hardcoded to only look in lib/, not also check lib64/.  If you install the cuda toolkit(s) via canonical NVidia packages and it’s an x86_64 box without multilib or compat for one of the 32bit variants, then don’t the libs go exclusively in /usr/lib64?</p>

username: ptrblck
cooked: <p>I don’t think that’s the case as my setup also doesn’t have the <code>lib</code> folder.<br>
Changing the location of the CUDA toolkit works fine, too:</p>
<pre><code class="lang-auto">ls /usr/local/cuda
NsightSystems-cli-2021.3.2  bin  compat  compute-sanitizer  extras  include  lib64  nvml  nvvm  share  src  targets

ln -s /usr/local/cuda ./mylocalcuda

CUDA_HOME=/workspace/src/mylocalcuda python setup.py develop
</code></pre>
<p>Output from the install log:</p>
<pre><code class="lang-auto">-- Caffe2: CUDA detected: 11.5
-- Caffe2: CUDA nvcc is: /workspace/src/mylocalcuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /workspace/src/mylocalcuda
...
--     CUDA version        : 11.5
...
--     CUDA root directory : /workspace/src/mylocalcuda
--     CUDA library        : /workspace/src/mylocalcuda/lib64/stubs/libcuda.so
--     cudart library      : /workspace/src/mylocalcuda/lib64/libcudart.so
--     cublas library      : /workspace/src/mylocalcuda/lib64/libcublas.so
--     cufft library       : /workspace/src/mylocalcuda/lib64/libcufft.so
--     curand library      : /workspace/src/mylocalcuda/lib64/libcurand.so
...
--     nvrtc               : /workspace/src/mylocalcuda/lib64/libnvrtc.so
--     CUDA include path   : /workspace/src/mylocalcuda/include
--     NVCC executable     : /workspace/src/mylocalcuda/bin/nvcc
</code></pre>

username: olagarde
cooked: <p>If I hardcode the</p>
<blockquote>
<pre><code class="lang-auto">set(CMAKE_CUDA_COMPILER_LIBRARY_ROOT "/opt/soft/cuda/11.4.2-1")
</code></pre>
</blockquote>
<p>in CMakeDetermineCUDACompiler.cmake to force it past that point just to see what breaks, it fails the nvcc “simple test program” precheck. Manually repeating that precheck succeeds.  I don’t get this unless the cuda install is relocated so… bad cuda relocation?  Both /usr/local and /opt/soft/cuda/{version}/ cuda installs successfully build and run their samples. Same thing if both installs coexist or are on different login node images, so it’s not an intersect. The 11.4.2-1 was prepackage, bet there’s something wrong with it’s install despite the successful samples. Either that or something with the combo of 11.4.2-1 and python 3.10.0 for the pytorch build?</p>
<p>What install method for your cuda, the one in /workspace/src? Tarball, “raw” installer, package, module, other?</p>
<p>What python?</p>

username: ptrblck
cooked: <p>I don’t think Python==3.10 is supported yet and the implementation is tracked <a href="https://github.com/pytorch/pytorch/issues/66424">here</a>.</p>
<aside class="quote no-group" data-username="olagarde" data-post="6" data-topic="136015">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/3bc359/48.png" class="avatar"> olagarde:</div>
<blockquote>
<p>What install method for your cuda, the one in /workspace/src? Tarball, “raw” installer, package, module, other?</p>
</blockquote>
</aside>
<p>Local installer via the <code>cuda_11...._linux.run</code> file, Python=3.8.</p>

username: olagarde
cooked: <p>Ah, right you are.  3.8.6 looks like the popular choice.  Right, modulefile for the python version, 3.10.0 for the other toolsets, 3.8.6 for pycuda, should do it.  Thanks!</p>