NanCheck.cu.o - ninja: build stopped: subcommand failed

Michael_Rowland: <p>Hello,</p>
<p>I’m trying to build pytorch from source for 2 x Tesla K80s CUDA11.4 but I get this error:</p>
<p>[7075/7977] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/torch/csrc/distributed/c10d/NanCheck.cu.o<br>
ninja: build stopped: subcommand failed.</p>
<p>I have:</p>
<p>NVIDIA-SMI 470.256.02   Driver Version: 470.256.02   CUDA Version: 11.4<br>
Cuda compilation tools, release 11.4, V11.4.100<br>
Build cuda_11.4.r11.4/compiler.30188945_0<br>
gcc (GCC) 9.3.0<br>
g++ (GCC) 9.3.0<br>
numpy 1.19.5<br>
Python 3.9.20<br>
Ninja version 1.11.1.git.kitware.jobserver-1</p>
<p>Tried with every gcc from 9.3.0 to 11th version and I get the same error.</p>
<p>Tried to solve this with both Copilot and Chatgpt but nothing works anymore. If anyone can offer his support I really appreciate it.</p>
<p>Thank you.</p>

ptrblck: <p>The build log should show a previous error which is causing <code>ninja</code> to fail, so search for any failures before the posted output.</p>

Michael_Rowland: <p>Hello,</p>
<p>Thank you so much for your reply.<br>
After a lot of hours trying to fix various errors I am back where I started with the same error.<br>
It stops at the same  error. Tried various cudnn versions, tried changing/adding things in the CMakeLists.txt file.</p>
<p>Nothing helped.</p>
<p>Can’t figure out what’s wrong with it. I would really really appreciate every help or direction I can receive. Some of the output on my screen is:</p>
<p>devtest01:~/pytorch$ python3 setup.py install<br>
Building wheel torch-2.6.0a0+gitd2ec289</p>
<p>– Building version 2.6.0a0+gitd2ec289</p>
<p>cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/mike/pytorch/torch -DCMAKE_PREFIX_PATH=/usr/lib/python3.9/site-packages;/usr -DPython_EXECUTABLE=/usr/bin/python3 -DTORCH_BUILD_VERSION=2.6.0a0+gitd2ec289 -DTORCH_CUDA_ARCH_LIST=3.7+PTX -DUSE_NUMPY=True /home/mike/pytorch</p>
<p>– The CXX compiler identification is GNU 9.3.0</p>
<p>– The C compiler identification is GNU 9.3.0</p>
<p>– Detecting CXX compiler ABI info</p>
<p>– Detecting CXX compiler ABI info - done</p>
<p>– Check for working CXX compiler: /usr/bin/c++ - skipped</p>
<p>– Detecting CXX compile features</p>
<p>– Detecting CXX compile features - done</p>
<p>– Detecting C compiler ABI info</p>
<p>– Detecting C compiler ABI info - done</p>
<p>– Check for working C compiler: /usr/bin/cc - skipped</p>
<p>– Detecting C compile features</p>
<p>– Detecting C compile features - done</p>
<p>– /usr/bin/c++ /home/mike/pytorch/torch/abi-check.cpp -o /home/mike/pytorch/build/abi-check</p>
<p>– Determined _GLIBCXX_USE_CXX11_ABI=1</p>
<p>– No SVE processor on this machine.</p>
<p>– Compiler does not support SVE extension. Will not build perfkernels.</p>
<p>– Found CUDA: /usr/local/cuda-11.4 (found version “11.4”)</p>
<p>– The CUDA compiler identification is NVIDIA 11.4.100</p>
<p>– Detecting CUDA compiler ABI info</p>
<p>– Detecting CUDA compiler ABI info - done</p>
<p>– Check for working CUDA compiler: /usr/local/cuda-11.4/bin/nvcc - skipped</p>
<p>– Detecting CUDA compile features</p>
<p>– Detecting CUDA compile features - done</p>
<p>– Found CUDAToolkit: /usr/local/cuda-11.4/include (found version “11.4.100”)</p>
<p>– Performing Test CMAKE_HAVE_LIBC_PTHREAD</p>
<p>– Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success</p>
<p>– Found Threads: TRUE</p>
<p>– Caffe2: CUDA detected: 11.4</p>
<p>– Caffe2: CUDA nvcc is: /usr/local/cuda-11.4/bin/nvcc</p>
<p>– Caffe2: CUDA toolkit directory: /usr/local/cuda-11.4</p>
<p>– Caffe2: Header version is: 11.4</p>
<p>– Found Python: /usr/bin/python3 (found version “3.9.20”) found components: Interpreter</p>
<p>CMake Warning at cmake/public/cuda.cmake:140 (message):</p>
<p>Failed to compute shorthash for libnvrtc.so</p>
<p>Call Stack (most recent call first):</p>
<p>cmake/Dependencies.cmake:44 (include)</p>
<p>CMakeLists.txt:853 (include)</p>
<p>– Found nvtx3: /home/mike/pytorch/third_party/NVTX/c/include</p>
<p>– Found CUDNN: /usr/local/cuda-11.4/lib64/libcudnn.so</p>
<p>– Found CUSPARSELT: /usr/lib/x86_64-linux-gnu/libcusparseLt.so</p>
<p>– Found CUDSS: /usr/lib/x86_64-linux-gnu/libcudss.so</p>
<p>– USE_CUFILE is set to 0. Compiling without cuFile support</p>
<p>– Added CUDA NVCC flags for: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_37,code=compute_37</p>
<p>CMake Warning at cmake/Dependencies.cmake:95 (message):</p>
<p>Not compiling with XPU. Could NOT find SYCL.Suppress this warning with</p>
<p>-DUSE_XPU=OFF.</p>
<p>Call Stack (most recent call first):</p>
<p>CMakeLists.txt:853 (include)</p>
<p>– Building using own protobuf under third_party per request.</p>
<p>– Use custom protobuf build.</p>
<p>CMake Deprecation Warning at third_party/protobuf/cmake/CMakeLists.txt:2 (cmake_minimum_required):</p>
<p>Compatibility with CMake &lt; 3.5 will be removed from a future version of</p>
<p>CMake.</p>
<p>Update the VERSION argument  value or use a … suffix to tell</p>
<p>CMake that the project does not need compatibility with older versions.</p>
<p>–</p>
<p>– 3.13.0.0</p>
<p>– Performing Test protobuf_HAVE_BUILTIN_ATOMICS</p>
<p>– Performing Test protobuf_HAVE_BUILTIN_ATOMICS - Success</p>
<p>– Caffe2 protobuf include directory: $&lt;BUILD_INTERFACE:/home/mike/pytorch/third_party/protobuf/src&gt;$&lt;INSTALL_INTERFACE:include&gt;</p>
<p>– Trying to find preferred BLAS backend of choice: MKL</p>
<p>– MKL_THREADING = OMP<br>
/home/mike/pytorch/aten/src/ATen/cuda/Exceptions.h:111:48: error: ‘cusolverStatus_t’ was not declared in this scope; did you mean ‘cusparseStatus_t’?<br>
111 | C10_EXPORT const char* cusolverGetErrorMessage(cusolverStatus_t status);<br>
|                                                ^~~~~~~~~~~~~~~~<br>
|                                                cusparseStatus_t<br>
[7052/7981] Building CXX object caffe2/CMakeFiles/torch_cuda.dir/<strong>/aten/src/ATen/native/quantized/cudnn/Linear.cpp.o<br>
FAILED: caffe2/CMakeFiles/torch_cuda.dir/</strong>/aten/src/ATen/native/quantized/cudnn/Linear.cpp.o<br>
/usr/bin/ccache /usr/bin/c++ -DAT_PER_OPERATOR_HEADERS -DFLASHATTENTION_DISABLE_ALIBI -DFMT_HEADER_ONLY=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DIDEEP_USE_MKL -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DTORCH_CUDA_BUILD_MAIN_LIB -DTORCH_CUDA_USE_NVTX3 -DUSE_C10D_GLOO -DUSE_C10D_NCCL -DUSE_CUDA -DUSE_CUDSS -DUSE_CUSPARSELT -DUSE_DISTRIBUTED -DUSE_EXTERNAL_MZCRC -DUSE_FLASH_ATTENTION -DUSE_MEM_EFF_ATTENTION -DUSE_NCCL -DUSE_RPC -DUSE_TENSORPIPE -D_FILE_OFFSET_BITS=64 -Dtorch_cuda_EXPORTS -I/home/mike/pytorch/build/aten/src -I/home/mike/pytorch/aten/src -I/home/mike/pytorch/build -I/home/mike/pytorch -I/home/mike/pytorch/cmake/…/third_party/benchmark/include -I/home/mike/pytorch/third_party/onnx -I/home/mike/pytorch/build/third_party/onnx -I/home/mike/pytorch/nlohmann -I/home/mike/pytorch/aten/src/THC -I/home/mike/pytorch/aten/src/ATen/cuda -I/home/mike/pytorch/third_party/fmt/include -I/home/mike/pytorch/aten/src/ATen/…/…/…/third_party/cutlass/include -I/home/mike/pytorch/aten/src/ATen/…/…/…/third_party/cutlass/tools/util/include -I/home/mike/pytorch/build/caffe2/aten/src -I/home/mike/pytorch/aten/src/ATen/… -I/home/mike/pytorch/build/nccl/include -I/home/mike/pytorch/c10/cuda/…/… -I/home/mike/pytorch/c10/… -I/home/mike/pytorch/third_party/tensorpipe -I/home/mike/pytorch/build/third_party/tensorpipe -I/home/mike/pytorch/third_party/tensorpipe/third_party/libnop/include -I/home/mike/pytorch/torch/csrc/api -I/home/mike/pytorch/torch/csrc/api/include -isystem /home/mike/pytorch/build/third_party/gloo -isystem /home/mike/pytorch/cmake/…/third_party/gloo -isystem /home/mike/pytorch/cmake/…/third_party/tensorpipe/third_party/libuv/include -isystem /home/mike/pytorch/cmake/…/third_party/googletest/googlemock/include -isystem /home/mike/pytorch/cmake/…/third_party/googletest/googletest/include -isystem /home/mike/pytorch/third_party/protobuf/src -isystem /usr/include/mkl -isystem /home/mike/pytorch/third_party/XNNPACK/include -isystem /home/mike/pytorch/third_party/ittapi/include -isystem /home/mike/pytorch/cmake/…/third_party/eigen -isystem /usr/local/cuda-11.4/include -isystem /home/mike/pytorch/third_party/ideep/mkl-dnn/include/oneapi/dnnl -isystem /home/mike/pytorch/third_party/ideep/include -isystem /home/mike/pytorch/INTERFACE -isystem /home/mike/pytorch/third_party/nlohmann/include -isystem /home/mike/pytorch/third_party/NVTX/c/include -isystem /home/mike/pytorch/cmake/…/third_party/cudnn_frontend/include -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -std=gnu++17 -fPIC -DMKL_HAS_SBGEMM -DTORCH_USE_LIBUV -DCAFFE2_USE_GLOO -Wall -Wextra -Wdeprecated -Wno-unused-parameter -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-strict-overflow -Wno-strict-aliasing -Wunused-function -Wunused-variable -Wunused-but-set-variable -Wno-maybe-uninitialized -fvisibility=hidden -O2 -MD -MT caffe2/CMakeFiles/torch_cuda.dir/<strong>/aten/src/ATen/native/quantized/cudnn/Linear.cpp.o -MF caffe2/CMakeFiles/torch_cuda.dir/</strong>/aten/src/ATen/native/quantized/cudnn/Linear.cpp.o.d -o caffe2/CMakeFiles/torch_cuda.dir/<strong>/aten/src/ATen/native/quantized/cudnn/Linear.cpp.o -c /home/mike/pytorch/aten/src/ATen/native/quantized/cudnn/Linear.cpp<br>
In file included from /home/mike/pytorch/aten/src/ATen/native/quantized/cudnn/Linear.cpp:9:<br>
/home/mike/pytorch/aten/src/ATen/cuda/Exceptions.h:111:48: error: ‘cusolverStatus_t’ was not declared in this scope; did you mean ‘cusparseStatus_t’?<br>
111 | C10_EXPORT const char* cusolverGetErrorMessage(cusolverStatus_t status);<br>
|                                                ^~~~~~~~~~~~~~~~<br>
|                                                cusparseStatus_t<br>
[7055/7981] Building CXX object caffe2/CMakeFiles/torch_cuda.dir/</strong>/aten/src/ATen/native/quantized/cudnn/Conv.cpp.o<br>
FAILED: caffe2/CMakeFiles/torch_cuda.dir/<strong>/aten/src/ATen/native/quantized/cudnn/Conv.cpp.o<br>
/usr/bin/ccache /usr/bin/c++ -DAT_PER_OPERATOR_HEADERS -DFLASHATTENTION_DISABLE_ALIBI -DFMT_HEADER_ONLY=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DIDEEP_USE_MKL -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DTORCH_CUDA_BUILD_MAIN_LIB -DTORCH_CUDA_USE_NVTX3 -DUSE_C10D_GLOO -DUSE_C10D_NCCL -DUSE_CUDA -DUSE_CUDSS -DUSE_CUSPARSELT -DUSE_DISTRIBUTED -DUSE_EXTERNAL_MZCRC -DUSE_FLASH_ATTENTION -DUSE_MEM_EFF_ATTENTION -DUSE_NCCL -DUSE_RPC -DUSE_TENSORPIPE -D_FILE_OFFSET_BITS=64 -Dtorch_cuda_EXPORTS -I/home/mike/pytorch/build/aten/src -I/home/mike/pytorch/aten/src -I/home/mike/pytorch/build -I/home/mike/pytorch -I/home/mike/pytorch/cmake/…/third_party/benchmark/include -I/home/mike/pytorch/third_party/onnx -I/home/mike/pytorch/build/third_party/onnx -I/home/mike/pytorch/nlohmann -I/home/mike/pytorch/aten/src/THC -I/home/mike/pytorch/aten/src/ATen/cuda -I/home/mike/pytorch/third_party/fmt/include -I/home/mike/pytorch/aten/src/ATen/…/…/…/third_party/cutlass/include -I/home/mike/pytorch/aten/src/ATen/…/…/…/third_party/cutlass/tools/util/include -I/home/mike/pytorch/build/caffe2/aten/src -I/home/mike/pytorch/aten/src/ATen/… -I/home/mike/pytorch/build/nccl/include -I/home/mike/pytorch/c10/cuda/…/… -I/home/mike/pytorch/c10/… -I/home/mike/pytorch/third_party/tensorpipe -I/home/mike/pytorch/build/third_party/tensorpipe -I/home/mike/pytorch/third_party/tensorpipe/third_party/libnop/include -I/home/mike/pytorch/torch/csrc/api -I/home/mike/pytorch/torch/csrc/api/include -isystem /home/mike/pytorch/build/third_party/gloo -isystem /home/mike/pytorch/cmake/…/third_party/gloo -isystem /home/mike/pytorch/cmake/…/third_party/tensorpipe/third_party/libuv/include -isystem /home/mike/pytorch/cmake/…/third_party/googletest/googlemock/include -isystem /home/mike/pytorch/cmake/…/third_party/googletest/googletest/include -isystem /home/mike/pytorch/third_party/protobuf/src -isystem /usr/include/mkl -isystem /home/mike/pytorch/third_party/XNNPACK/include -isystem /home/mike/pytorch/third_party/ittapi/include -isystem /home/mike/pytorch/cmake/…/third_party/eigen -isystem /usr/local/cuda-11.4/include -isystem /home/mike/pytorch/third_party/ideep/mkl-dnn/include/oneapi/dnnl -isystem /home/mike/pytorch/third_party/ideep/include -isystem /home/mike/pytorch/INTERFACE -isystem /home/mike/pytorch/third_party/nlohmann/include -isystem /home/mike/pytorch/third_party/NVTX/c/include -isystem /home/mike/pytorch/cmake/…/third_party/cudnn_frontend/include -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -std=gnu++17 -fPIC -DMKL_HAS_SBGEMM -DTORCH_USE_LIBUV -DCAFFE2_USE_GLOO -Wall -Wextra -Wdeprecated -Wno-unused-parameter -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-strict-overflow -Wno-strict-aliasing -Wunused-function -Wunused-variable -Wunused-but-set-variable -Wno-maybe-uninitialized -fvisibility=hidden -O2 -MD -MT caffe2/CMakeFiles/torch_cuda.dir/</strong>/aten/src/ATen/native/quantized/cudnn/Conv.cpp.o -MF caffe2/CMakeFiles/torch_cuda.dir/<strong>/aten/src/ATen/native/quantized/cudnn/Conv.cpp.o.d -o caffe2/CMakeFiles/torch_cuda.dir/</strong>/aten/src/ATen/native/quantized/cudnn/Conv.cpp.o -c /home/mike/pytorch/aten/src/ATen/native/quantized/cudnn/Conv.cpp<br>
In file included from /home/mike/pytorch/aten/src/ATen/native/quantized/cudnn/Conv.cpp:9:<br>
/home/mike/pytorch/aten/src/ATen/cuda/Exceptions.h:111:48: error: ‘cusolverStatus_t’ was not declared in this scope; did you mean ‘cusparseStatus_t’?<br>
111 | C10_EXPORT const char* cusolverGetErrorMessage(cusolverStatus_t status);<br>
|                                                ^~~~~~~~~~~~~~~~<br>
|                                                cusparseStatus_t<br>
[7057/7981] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/torch/csrc/distributed/c10d/CUDASymmetricMemory.cu.o<br>
/home/mike/pytorch/torch/csrc/distributed/c10d/CUDASymmetricMemory.cu(25): warning: parameter “device_idx” was declared but never referenced</p>
<p>/home/mike/pytorch/torch/csrc/distributed/c10d/CUDASymmetricMemory.cu(154): warning: function “::IpcChannel::broadcast_fds” was declared but never referenced</p>
<p>[7063/7981] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/torch/csrc/distributed/c10d/NanCheck.cu.o<br>
ninja: build stopped: subcommand failed.</p>

ptrblck: <p>The actual failure is:</p>
<pre><code class="lang-auto">/home/mike/pytorch/aten/src/ATen/cuda/Exceptions.h:111:48: error: ‘cusolverStatus_t’ was not declared in this scope; did you mean ‘cusparseStatus_t’?
111 | C10_EXPORT const char* cusolverGetErrorMessage(cusolverStatus_t status);
| ^~~~~~~~~~~~~~~~
| cusparseStatus_t
</code></pre>
<p>which points to a missing <code>cuSOLVER</code> library. Check if <code>cuSOLVER</code> is installed and if not, reinstall your CUDA toolkit with it.</p>