Ninja: build stopped: subcommand failed. when installing pytorch from source

thomasB: <p>Hi all,<br>
I’m trying to install pytorch from source, but I’m getting the “ninja: build stopped: subcomman failed. error”. Here are the versions I’m using:<br>
Ubuntu: 16.04.5 LTS<br>
GCC: 5.4.0<br>
CUDA: 11.0<br>
CUDNN: 8.1.0<br>
Python: 3.8.8<br>
pytorch: latest, just cloned from git<br>
The error message I get is<br>
[2708/6039] Building CXX object third_party/ideep/mkl-dnn/src/cpu/CMakeFiles/dnnl_cpu.dir/cpu_reorder.cpp.o<br>
ninja: build stopped: subcommand failed.<br>
Traceback (most recent call last):<br>
File “setup.py”, line 876, in <br>
build_deps()<br>
File “setup.py”, line 354, in build_deps<br>
build_caffe2(version=version,<br>
File “/local/scratch/public/pytorch/tools/build_pytorch_libs.py”, line 58, in build_caffe2<br>
cmake.build(my_env)<br>
File “/local/scratch/public/pytorch/tools/setup_helpers/cmake.py”, line 345, in build<br>
self.run(build_args, my_env)<br>
File “/local/scratch/public/pytorch/tools/setup_helpers/cmake.py”, line 140, in run<br>
check_call(command, cwd=self.build_dir, env=env)<br>
File “/alt/applic/user-maint/tb588/Conda/envs/pytorch_env/lib/python3.8/subprocess.py”, line 364, in check_call<br>
raise CalledProcessError(retcode, cmd)<br>
subprocess.CalledProcessError: Command ‘[‘cmake’, ‘–build’, ‘.’, ‘–target’, ‘install’, ‘–config’, ‘Release’, ‘–’, ‘-j’, ‘48’]’ returned non-zero exit status 1.<br>
I tried</p>
<pre><code class="lang-auto">git clean -xdf
python setup.py clean
git submodule sync
git submodule deinit -f .
git submodule update --init --recursive
python setup.py install
</code></pre>
<p>but that didn’t help. Does anyone know how to fix this?<br>
Thanks!</p>

ptrblck: <p>The error message should have been printed in the install log before ninja is failing, so you would have to recheck the log.<br>
In case you cannot fix the error, feel free to post it here and we can have a look.</p>

thomasB: <p>Thanks! Is there an install log that is automatically created when installing pytorch from source?<br>
When I tried the installation I used</p>
<p>python setup.py install &gt; install_log.txt</p>
<p>The last few lines in this document are</p>
<p>[2703/6039] Building CXX object third_party/ideep/mkl-dnn/src/cpu/CMakeFiles/dnnl_cpu.dir/rnn/ref_rnn.cpp.o<br>
[2704/6039] Building CXX object third_party/fbgemm/CMakeFiles/fbgemm_avx2.dir/src/FbgemmI8DepthwiseAvx2.cc.o<br>
[2705/6039] Building CXX object third_party/fbgemm/CMakeFiles/fbgemm_avx2.dir/src/FbgemmI8Depthwise3DAvx2.cc.o<br>
[2706/6039] Building CXX object third_party/ideep/mkl-dnn/src/cpu/CMakeFiles/dnnl_cpu.dir/cpu_reorder.cpp.o<br>
ninja: build stopped: subcommand failed.<br>
Building wheel torch-1.9.0a0+git09756e7<br>
– Building version 1.9.0a0+git09756e7<br>
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/local/scratch/public/pytorch/torch -DCMAKE_PREFIX_PATH=/alt/applic/user-maint/tb588/Conda/envs/pytorch_env -DCUDNN_INCLUDE_DIR=/local/scratch/public/cudnn/cuda/include -DCUDNN_LIBRARY=/local/scratch/public/cudnn/cuda/lib64 -DNUMPY_INCLUDE_DIR=/alt/applic/user-maint/tb588/Conda/envs/pytorch_env/lib/python3.8/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/alt/applic/user-maint/tb588/Conda/envs/pytorch_env/bin/python -DPYTHON_INCLUDE_DIR=/alt/applic/user-maint/tb588/Conda/envs/pytorch_env/include/python3.8 -DPYTHON_LIBRARY=/alt/applic/user-maint/tb588/Conda/envs/pytorch_env/lib/libpython3.8.so.1.0 -DTORCH_BUILD_VERSION=1.9.0a0+git09756e7 -DUSE_NUMPY=True /local/scratch/public/pytorch<br>
cmake --build . --target install --config Release – -j 48</p>
<p>Thanks for the help!</p>

CLaireeeee: <p>I also had this problem for a long time while installing on linux and have no idea how to deal with it. Any help will be appreciated!</p>
<p>Here is my console output:</p>
<pre><code class="lang-auto">Building wheel torch-1.9.0a0+git55dfb4a

-- Building version 1.9.0a0+git55dfb4a

cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/caowanlu/fnet/pytorch/torch -DCMAKE_PREFIX_PATH=/usr/lib/python3/dist-packages -DNUMPY_INCLUDE_DIR=/home/caowanlu/.local/lib/python3.6/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/usr/bin/python3 -DPYTHON_INCLUDE_DIR=/usr/include/python3.6m -DPYTHON_LIBRARY=/usr/lib/libpython3.6m.so.1.0 -DTORCH_BUILD_VERSION=1.9.0a0+git55dfb4a -DUSE_NUMPY=True /home/caowanlu/fnet/pytorch

-- The CXX compiler identification is GNU 7.5.0

-- The C compiler identification is GNU 7.5.0

-- Detecting CXX compiler ABI info

-- Detecting CXX compiler ABI info - done

-- Check for working CXX compiler: /usr/bin/c++ - skipped

-- Detecting CXX compile features

-- Detecting CXX compile features - done

-- Detecting C compiler ABI info

-- Detecting C compiler ABI info - done

-- Check for working C compiler: /usr/bin/cc - skipped

-- Detecting C compile features

-- Detecting C compile features - done

-- Not forcing any particular BLAS to be found

-- Performing Test COMPILER_WORKS

-- Performing Test COMPILER_WORKS - Success

-- Performing Test SUPPORT_GLIBCXX_USE_C99

-- Performing Test SUPPORT_GLIBCXX_USE_C99 - Success

-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED

-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED - Success

-- std::exception_ptr is supported.

-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING

-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING - Failed

-- Turning off deprecation warning due to glog.

-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS

-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS - Success

-- Current compiler supports avx2 extension. Will build perfkernels.

-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS

-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS - Success

-- Current compiler supports avx512f extension. Will build fbgemm.

-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY

-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY - Success

-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY

-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY - Success

-- Performing Test COMPILER_SUPPORTS_RDYNAMIC

-- Performing Test COMPILER_SUPPORTS_RDYNAMIC - Success

-- Building using own protobuf under third_party per request.

-- Use custom protobuf build.

--

-- 3.11.4.0

-- Looking for pthread.h

-- Looking for pthread.h - found

-- Performing Test CMAKE_HAVE_LIBC_PTHREAD

-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed

-- Check if compiler accepts -pthread

-- Check if compiler accepts -pthread - yes

-- Found Threads: TRUE

-- Performing Test protobuf_HAVE_BUILTIN_ATOMICS

-- Performing Test protobuf_HAVE_BUILTIN_ATOMICS - Success

-- Caffe2 protobuf include directory: $&lt;BUILD_INTERFACE:/home/caowanlu/fnet/pytorch/third_party/protobuf/src&gt;$&lt;INSTALL_INTERFACE:include&gt;

-- Trying to find preferred BLAS backend of choice: MKL

-- MKL_THREADING = OMP

-- Looking for sys/types.h

-- Looking for sys/types.h - found

-- Looking for stdint.h

-- Looking for stdint.h - found

-- Looking for stddef.h

-- Looking for stddef.h - found

-- Check size of void*

-- Check size of void* - done

CMake Warning (dev) at /home/caowanlu/anaconda3/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):

The package name passed to `find_package_handle_standard_args` (OpenMP_C)

does not match the name of the calling package (OpenMP). This can lead to

problems in calling code that expects `find_package` result variables

(e.g., `_FOUND`) to follow a certain pattern.

Call Stack (most recent call first):

cmake/Modules/FindOpenMP.cmake:576 (find_package_handle_standard_args)

cmake/Modules/FindMKL.cmake:213 (FIND_PACKAGE)

cmake/Modules/FindMKL.cmake:307 (CHECK_ALL_LIBRARIES)

cmake/Dependencies.cmake:140 (find_package)

CMakeLists.txt:615 (include)

This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at /home/caowanlu/anaconda3/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):

The package name passed to `find_package_handle_standard_args` (OpenMP_CXX)

does not match the name of the calling package (OpenMP). This can lead to

problems in calling code that expects `find_package` result variables

(e.g., `_FOUND`) to follow a certain pattern.

Call Stack (most recent call first):

cmake/Modules/FindOpenMP.cmake:576 (find_package_handle_standard_args)

cmake/Modules/FindMKL.cmake:213 (FIND_PACKAGE)

cmake/Modules/FindMKL.cmake:307 (CHECK_ALL_LIBRARIES)

cmake/Dependencies.cmake:140 (find_package)

CMakeLists.txt:615 (include)

This warning is for project developers. Use -Wno-dev to suppress it.

-- Looking for cblas_sgemm

-- Looking for cblas_sgemm - found

-- MKL libraries: /home/caowanlu/anaconda3/lib/libmkl_intel_lp64.so;/home/caowanlu/anaconda3/lib/libmkl_gnu_thread.so;/home/caowanlu/anaconda3/lib/libmkl_core.so;-fopenmp;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libm.so;/usr/lib/x86_64-linux-gnu/libdl.so

-- MKL include directory: /home/caowanlu/anaconda3/include

-- MKL OpenMP type: GNU

-- MKL OpenMP library: -fopenmp

-- The ASM compiler identification is GNU

-- Found assembler: /usr/bin/cc

-- Brace yourself, we are building NNPACK

-- Performing Test NNPACK_ARCH_IS_X86_32

-- Performing Test NNPACK_ARCH_IS_X86_32 - Failed

-- Found PythonInterp: /usr/bin/python3 (found version "3.6.9")

-- NNPACK backend is x86-64

CMake Deprecation Warning at third_party/googletest/CMakeLists.txt:1 (cmake_minimum_required):

Compatibility with CMake &lt; 2.8.12 will be removed from a future version of

CMake.

Update the VERSION argument &lt;min&gt; value or use a ...&lt;max&gt; suffix to tell

CMake that the project does not need compatibility with older versions.

CMake Deprecation Warning at third_party/googletest/googlemock/CMakeLists.txt:42 (cmake_minimum_required):

Compatibility with CMake &lt; 2.8.12 will be removed from a future version of

CMake.

Update the VERSION argument &lt;min&gt; value or use a ...&lt;max&gt; suffix to tell

CMake that the project does not need compatibility with older versions.

CMake Deprecation Warning at third_party/googletest/googletest/CMakeLists.txt:49 (cmake_minimum_required):

Compatibility with CMake &lt; 2.8.12 will be removed from a future version of

CMake.

Update the VERSION argument &lt;min&gt; value or use a ...&lt;max&gt; suffix to tell

CMake that the project does not need compatibility with older versions.

-- Failed to find LLVM FileCheck

-- Found Git: /usr/bin/git (found version "2.17.1")

-- git Version: v1.4.0-505be96a-dirty

-- Version: 1.4.0

-- Performing Test HAVE_CXX_FLAG_STD_CXX11

-- Performing Test HAVE_CXX_FLAG_STD_CXX11 - Success

-- Performing Test HAVE_CXX_FLAG_WALL

-- Performing Test HAVE_CXX_FLAG_WALL - Success

-- Performing Test HAVE_CXX_FLAG_LPTHREAD

-- Performing Test HAVE_CXX_FLAG_LPTHREAD - Success

-- Performing Test HAVE_CXX_FLAG_WEXTRA

-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success

-- Performing Test HAVE_CXX_FLAG_WSHADOW

-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success

-- Performing Test HAVE_CXX_FLAG_WERROR

-- Performing Test HAVE_CXX_FLAG_WERROR - Success

-- Performing Test HAVE_CXX_FLAG_PEDANTIC

-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success

-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS

-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success

-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL

-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success

-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING

-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success

-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS

-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS - Success

-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING

-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success

-- Performing Test HAVE_CXX_FLAG_COVERAGE

-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success

-- Performing Test HAVE_STD_REGEX

-- Performing Test HAVE_STD_REGEX

-- Performing Test HAVE_STD_REGEX -- success

-- Performing Test HAVE_GNU_POSIX_REGEX

-- Performing Test HAVE_GNU_POSIX_REGEX

-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile

-- Performing Test HAVE_POSIX_REGEX

-- Performing Test HAVE_POSIX_REGEX

-- Performing Test HAVE_POSIX_REGEX -- success

-- Performing Test HAVE_STEADY_CLOCK

-- Performing Test HAVE_STEADY_CLOCK

-- Performing Test HAVE_STEADY_CLOCK -- success

-- Performing Test COMPILER_SUPPORTS_AVX512

-- Performing Test COMPILER_SUPPORTS_AVX512 - Success

CMake Warning (dev) at /home/caowanlu/anaconda3/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):

The package name passed to `find_package_handle_standard_args` (OpenMP_C)

does not match the name of the calling package (OpenMP). This can lead to

problems in calling code that expects `find_package` result variables

(e.g., `_FOUND`) to follow a certain pattern.

Call Stack (most recent call first):

cmake/Modules/FindOpenMP.cmake:576 (find_package_handle_standard_args)

third_party/fbgemm/CMakeLists.txt:60 (find_package)

This warning is for project developers. Use -Wno-dev to suppress it.

-- Found OpenMP_C: -fopenmp (found version "4.5")

CMake Warning (dev) at /home/caowanlu/anaconda3/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):

The package name passed to `find_package_handle_standard_args` (OpenMP_CXX)

does not match the name of the calling package (OpenMP). This can lead to

problems in calling code that expects `find_package` result variables

(e.g., `_FOUND`) to follow a certain pattern.

Call Stack (most recent call first):

cmake/Modules/FindOpenMP.cmake:576 (find_package_handle_standard_args)

third_party/fbgemm/CMakeLists.txt:60 (find_package)

This warning is for project developers. Use -Wno-dev to suppress it.

-- Found OpenMP_CXX: -fopenmp (found version "4.5")

-- Found OpenMP: TRUE (found version "4.5")

CMake Warning at third_party/fbgemm/CMakeLists.txt:62 (message):

OpenMP found! OpenMP_C_INCLUDE_DIRS =

CMake Warning at third_party/fbgemm/CMakeLists.txt:142 (message):

==========

CMake Warning at third_party/fbgemm/CMakeLists.txt:143 (message):

CMAKE_BUILD_TYPE = Release

CMake Warning at third_party/fbgemm/CMakeLists.txt:144 (message):

CMAKE_CXX_FLAGS_DEBUG is -g

CMake Warning at third_party/fbgemm/CMakeLists.txt:145 (message):

CMAKE_CXX_FLAGS_RELEASE is -O3 -DNDEBUG

CMake Warning at third_party/fbgemm/CMakeLists.txt:146 (message):

==========

-- Performing Test __CxxFlag__fno_threadsafe_statics

-- Performing Test __CxxFlag__fno_threadsafe_statics - Success

-- Performing Test __CxxFlag__fno_semantic_interposition

-- Performing Test __CxxFlag__fno_semantic_interposition - Success

-- Performing Test __CxxFlag__fmerge_all_constants

-- Performing Test __CxxFlag__fmerge_all_constants - Success

-- Performing Test __CxxFlag__fno_enforce_eh_specs

-- Performing Test __CxxFlag__fno_enforce_eh_specs - Success

** AsmJit Summary **

ASMJIT_DIR=/home/caowanlu/fnet/pytorch/third_party/fbgemm/third_party/asmjit

ASMJIT_TEST=FALSE

ASMJIT_TARGET_TYPE=STATIC

ASMJIT_DEPS=pthread;rt

ASMJIT_LIBS=asmjit;pthread;rt

ASMJIT_CFLAGS=-DASMJIT_STATIC

ASMJIT_PRIVATE_CFLAGS=-Wall;-Wextra;-Wconversion;-fno-math-errno;-fno-threadsafe-statics;-fno-semantic-interposition;-DASMJIT_STATIC

ASMJIT_PRIVATE_CFLAGS_DBG=

ASMJIT_PRIVATE_CFLAGS_REL=-O2;-fmerge-all-constants;-fno-enforce-eh-specs

-- Found Numa: /usr/include

-- Found Numa (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libnuma.so)

-- Using third party subdirectory Eigen.

-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.9", minimum required is "3.0")

-- Found PythonLibs: /usr/lib/libpython3.6m.so.1.0 (found suitable version "3.6.9", minimum required is "3.0")

-- Could NOT find pybind11 (missing: pybind11_DIR)

-- Could NOT find pybind11 (missing: pybind11_INCLUDE_DIR)

-- Using third_party/pybind11.

-- pybind11 include dirs: /home/caowanlu/fnet/pytorch/cmake/../third_party/pybind11/include

-- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found version "3.1")

-- Found MPI_CXX: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so (found version "3.1")

-- Found MPI: TRUE (found version "3.1")

-- MPI support found

-- MPI compile flags: -pthread

-- MPI include path: /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include/usr/lib/x86_64-linux-gnu/openmpi/include

-- MPI LINK flags path: -L/usr/lib -pthread

-- MPI libraries: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so/usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so

CMake Warning at cmake/Dependencies.cmake:1040 (message):

OpenMPI found, but it is not built with CUDA support.

Call Stack (most recent call first):

CMakeLists.txt:615 (include)

CMake Warning (dev) at /home/caowanlu/anaconda3/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):

The package name passed to `find_package_handle_standard_args` (OpenMP_C)

does not match the name of the calling package (OpenMP). This can lead to

problems in calling code that expects `find_package` result variables

(e.g., `_FOUND`) to follow a certain pattern.

Call Stack (most recent call first):

cmake/Modules/FindOpenMP.cmake:576 (find_package_handle_standard_args)

cmake/Dependencies.cmake:1100 (find_package)

CMakeLists.txt:615 (include)

This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at /home/caowanlu/anaconda3/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):

The package name passed to `find_package_handle_standard_args` (OpenMP_CXX)

does not match the name of the calling package (OpenMP). This can lead to

problems in calling code that expects `find_package` result variables

(e.g., `_FOUND`) to follow a certain pattern.

Call Stack (most recent call first):

cmake/Modules/FindOpenMP.cmake:576 (find_package_handle_standard_args)

cmake/Dependencies.cmake:1100 (find_package)

CMakeLists.txt:615 (include)

This warning is for project developers. Use -Wno-dev to suppress it.

-- Adding OpenMP CXX_FLAGS: -fopenmp

-- Will link against OpenMP libraries: /usr/lib/gcc/x86_64-linux-gnu/7/libgomp.so;/usr/lib/x86_64-linux-gnu/libpthread.so

-- Found CUDA: /usr/local/cuda (found version "11.0")

-- Caffe2: CUDA detected: 11.0

-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc

-- Caffe2: CUDA toolkit directory: /usr/local/cuda

-- Caffe2: Header version is: 11.0

-- Found CUDNN: /usr/local/cuda/lib64/libcudnn.so

-- Found cuDNN: v8.1.0 (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)

-- /usr/local/cuda/lib64/libnvrtc.so shorthash is f4909b87

-- Autodetected CUDA architecture(s): 7.5

-- Added CUDA NVCC flags for: -gencode;arch=compute_75,code=sm_75

-- Found HIP: /opt/rocm-4.0.0/hip (found suitable version "4.0.20496-4f163c68", minimum required is "1.0")

HIP VERSION: 4.0.20496-4f163c68

***** ROCm version from /opt/rocm/.info/version-dev ****

ROCM_VERSION_DEV: 4.0.0

ROCM_VERSION_DEV_MAJOR: 4

ROCM_VERSION_DEV_MINOR: 0

ROCM_VERSION_DEV_PATCH: 0

***** Library versions from dpkg *****

rocm-dev VERSION: 4.0.0.40000-23

rocm-device-libs VERSION: 1.0.0.637-rocm-rel-4.0-23-db8c0c3

hsakmt-roct VERSION: 20201016.1.0269-mainline-20201016-1-g0269ce3

hsakmt-roct-dev VERSION: 20201016.1.0269-mainline-20201016-1-g0269ce3

hsa-rocr-dev VERSION: 1.2.40000.0-rocm-rel-4.0-23-a5173c90

***** Library versions from cmake find_package *****

hip VERSION: 4.0.20496

hsa-runtime64 VERSION: 1.2.40000

amd_comgr VERSION: 1.9.0.194-rocm-rel-4.0-23-0fa438b

CMake Error at cmake/public/LoadHIP.cmake:138 (find_package):

By not providing "Findrocrand.cmake" in CMAKE_MODULE_PATH this project has

asked CMake to find a package configuration file provided by "rocrand", but

CMake did not find one.

Could not find a package configuration file provided by "rocrand" with any

of the following names:

rocrandConfig.cmake

rocrand-config.cmake

Add the installation prefix of "rocrand" to CMAKE_PREFIX_PATH or set

"rocrand_DIR" to a directory containing one of the above files. If

"rocrand" provides a separate development package or SDK, be sure it has

been installed.

Call Stack (most recent call first):

cmake/public/LoadHIP.cmake:197 (find_package_and_print_version)

cmake/Dependencies.cmake:1189 (include)

CMakeLists.txt:615 (include)

-- Configuring incomplete, errors occurred!

See also "/home/caowanlu/fnet/pytorch/build/CMakeFiles/CMakeOutput.log".

See also "/home/caowanlu/fnet/pytorch/build/CMakeFiles/CMakeError.log".

Traceback (most recent call last):

File "setup.py", line 876, in &lt;module&gt;

build_deps()

File "setup.py", line 359, in build_deps

cmake=cmake)

File "/home/caowanlu/fnet/pytorch/tools/build_pytorch_libs.py", line 55, in build_caffe2

rerun_cmake)

File "/home/caowanlu/fnet/pytorch/tools/setup_helpers/cmake.py", line 329, in generate

self.run(args, env=my_env)

File "/home/caowanlu/fnet/pytorch/tools/setup_helpers/cmake.py", line 140, in run

check_call(command, cwd=self.build_dir, env=env)

File "/usr/lib/python3.6/subprocess.py", line 311, in check_call

raise CalledProcessError(retcode, cmd)

subprocess.CalledProcessError: Command '['cmake', '-GNinja', '-DBUILD_PYTHON=True', '-DBUILD_TEST=True', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/home/caowanlu/fnet/pytorch/torch', '-DCMAKE_PREFIX_PATH=/usr/lib/python3/dist-packages', '-DNUMPY_INCLUDE_DIR=/home/caowanlu/.local/lib/python3.6/site-packages/numpy/core/include', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DPYTHON_INCLUDE_DIR=/usr/include/python3.6m', '-DPYTHON_LIBRARY=/usr/lib/libpython3.6m.so.1.0', '-DTORCH_BUILD_VERSION=1.9.0a0+git55dfb4a', '-DUSE_NUMPY=True', '/home/caowanlu/fnet/pytorch']' returned non-zero exit status 1.</code></pre>

ptrblck: <p>The error might be earlier than the last couple of lines and you could <code>grep</code> for it or alternatively post the complete log (or the last part at least) as a code snippet here.</p>
<p><a class="mention" href="/u/claireeeee">@CLaireeeee</a><br>
The build seems to fail with:</p>
<pre><code class="lang-python">CMake Error at cmake/public/LoadHIP.cmake:138 (find_package):

By not providing “Findrocrand.cmake” in CMAKE_MODULE_PATH this project has

asked CMake to find a package configuration file provided by “rocrand”, but

CMake did not find one.

Could not find a package configuration file provided by “rocrand” with any

of the following names:

rocrandConfig.cmake

rocrand-config.cmake

Add the installation prefix of “rocrand” to CMAKE_PREFIX_PATH or set

“rocrand_DIR” to a directory containing one of the above files. If

“rocrand” provides a separate development package or SDK, be sure it has

been installed.

Call Stack (most recent call first):

cmake/public/LoadHIP.cmake:197 (find_package_and_print_version)

cmake/Dependencies.cmake:1189 (include)

CMakeLists.txt:615 (include)

– Configuring incomplete, errors occurred!
</code></pre>
<p>PS: you can post code snippets (or these types of logs) by wrapping them into three backticks ```, which would make debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

CLaireeeee: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your reply! I edited it and used ``` now.<br>
I saw the “rocrand” problem you mentioned and exported environment variable USE_ROCM=0 (as I really do not need it)</p>
<p><code> export USE_ROCM=0</code></p>
<p>Then the building continued!<br>
Thank you again!</p>

craigyang: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, when building from source, I also get the Ninja error in the title, but the message seems different from the above. Referring to <a href="https://github.com/pytorch/pytorch/issues/32694" class="inline-onebox" rel="noopener nofollow ugc">ninja: build stopped: subcommand failed. · Issue #32694 · pytorch/pytorch · GitHub</a>, I tried both the <code>git clean</code> code snippet (also posted in the question body) and <code>conda (or pip) install ninja</code>, but neither helped. Could you help take a look? Thanks!</p>
<p>Last several lines of output when I run <code>python setup.py develop</code> (with some environment variables set beforehand):</p>
<pre><code class="lang-auto">...
-- Generating done
-- Build files have been written to: /home/craig/pytorch/build
[17/381] Linking CXX executable bin/utility_ops_gpu_test
FAILED: bin/utility_ops_gpu_test 
: &amp;&amp; /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic -Wl,-rpath -Wl,/usr/local/openmpi/openmpi-4.1.0/lib -Wl,--enable-new-dtags -pthread caffe2/CMakeFiles/utility_ops_gpu_test.dir/operators/utility_ops_gpu_test.cc.o -o bin/utility_ops_gpu_test  -Wl,-rpath,/home/craig/anaconda3/envs/py/lib:/usr/local/cuda/lib64:/home/craig/pytorch/build/lib:  /usr/local/cuda/lib64/libcudart.so  lib/libgtest_main.a  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /home/craig/anaconda3/envs/py/lib/libmkl_intel_lp64.so  /home/craig/anaconda3/envs/py/lib/libmkl_gnu_thread.so  /home/craig/anaconda3/envs/py/lib/libmkl_core.so  -fopenmp  /usr/lib/x86_64-linux-gnu/libpthread.so  -lm  /usr/lib/x86_64-linux-gnu/libdl.so  lib/libdnnl.a  -ldl  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cuda.so" -Wl,--as-needed  lib/libc10_cuda.so  lib/libc10.so  /usr/local/cuda/lib64/libcudart.so  /home/craig/anaconda3/envs/py/lib/libnvToolsExt.so  /usr/local/cuda/lib64/libcufft.so  /usr/local/cuda/lib64/libcurand.so  /usr/local/cuda/lib64/libcublas.so  /usr/local/cuda/lib64/libcudnn.so  lib/libgtest.a  -pthread &amp;&amp; :
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmACC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTST@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHHH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmSSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBII@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
collect2: error: ld returned 1 exit status
[19/381] Linking CXX executable bin/generate_proposals_op_gpu_test
FAILED: bin/generate_proposals_op_gpu_test 
: &amp;&amp; /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic -Wl,-rpath -Wl,/usr/local/openmpi/openmpi-4.1.0/lib -Wl,--enable-new-dtags -pthread caffe2/CMakeFiles/generate_proposals_op_gpu_test.dir/operators/generate_proposals_op_gpu_test.cc.o -o bin/generate_proposals_op_gpu_test  -Wl,-rpath,/home/craig/anaconda3/envs/py/lib:/usr/local/cuda/lib64:/home/craig/pytorch/build/lib:  /usr/local/cuda/lib64/libcudart.so  lib/libgtest_main.a  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /home/craig/anaconda3/envs/py/lib/libmkl_intel_lp64.so  /home/craig/anaconda3/envs/py/lib/libmkl_gnu_thread.so  /home/craig/anaconda3/envs/py/lib/libmkl_core.so  -fopenmp  /usr/lib/x86_64-linux-gnu/libpthread.so  -lm  /usr/lib/x86_64-linux-gnu/libdl.so  lib/libdnnl.a  -ldl  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cuda.so" -Wl,--as-needed  lib/libc10_cuda.so  lib/libc10.so  /usr/local/cuda/lib64/libcudart.so  /home/craig/anaconda3/envs/py/lib/libnvToolsExt.so  /usr/local/cuda/lib64/libcufft.so  /usr/local/cuda/lib64/libcurand.so  /usr/local/cuda/lib64/libcublas.so  /usr/local/cuda/lib64/libcudnn.so  lib/libgtest.a  -pthread &amp;&amp; :
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmACC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTST@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHHH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmSSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBII@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
collect2: error: ld returned 1 exit status
[20/381] Linking CXX executable bin/roi_align_op_gpu_test
FAILED: bin/roi_align_op_gpu_test 
: &amp;&amp; /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic -Wl,-rpath -Wl,/usr/local/openmpi/openmpi-4.1.0/lib -Wl,--enable-new-dtags -pthread caffe2/CMakeFiles/roi_align_op_gpu_test.dir/operators/roi_align_op_gpu_test.cc.o -o bin/roi_align_op_gpu_test  -Wl,-rpath,/home/craig/anaconda3/envs/py/lib:/usr/local/cuda/lib64:/home/craig/pytorch/build/lib:  /usr/local/cuda/lib64/libcudart.so  lib/libgtest_main.a  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /home/craig/anaconda3/envs/py/lib/libmkl_intel_lp64.so  /home/craig/anaconda3/envs/py/lib/libmkl_gnu_thread.so  /home/craig/anaconda3/envs/py/lib/libmkl_core.so  -fopenmp  /usr/lib/x86_64-linux-gnu/libpthread.so  -lm  /usr/lib/x86_64-linux-gnu/libdl.so  lib/libdnnl.a  -ldl  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cuda.so" -Wl,--as-needed  lib/libc10_cuda.so  lib/libc10.so  /usr/local/cuda/lib64/libcudart.so  /home/craig/anaconda3/envs/py/lib/libnvToolsExt.so  /usr/local/cuda/lib64/libcufft.so  /usr/local/cuda/lib64/libcurand.so  /usr/local/cuda/lib64/libcublas.so  /usr/local/cuda/lib64/libcudnn.so  lib/libgtest.a  -pthread &amp;&amp; :
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmACC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTST@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHHH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmSSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBII@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
collect2: error: ld returned 1 exit status
[21/381] Linking CXX executable bin/batch_matmul_op_gpu_test
FAILED: bin/batch_matmul_op_gpu_test 
: &amp;&amp; /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic -Wl,-rpath -Wl,/usr/local/openmpi/openmpi-4.1.0/lib -Wl,--enable-new-dtags -pthread caffe2/CMakeFiles/batch_matmul_op_gpu_test.dir/operators/batch_matmul_op_gpu_test.cc.o -o bin/batch_matmul_op_gpu_test  -Wl,-rpath,/home/craig/anaconda3/envs/py/lib:/usr/local/cuda/lib64:/home/craig/pytorch/build/lib:  /usr/local/cuda/lib64/libcudart.so  lib/libgtest_main.a  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /home/craig/anaconda3/envs/py/lib/libmkl_intel_lp64.so  /home/craig/anaconda3/envs/py/lib/libmkl_gnu_thread.so  /home/craig/anaconda3/envs/py/lib/libmkl_core.so  -fopenmp  /usr/lib/x86_64-linux-gnu/libpthread.so  -lm  /usr/lib/x86_64-linux-gnu/libdl.so  lib/libdnnl.a  -ldl  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cuda.so" -Wl,--as-needed  lib/libc10_cuda.so  lib/libc10.so  /usr/local/cuda/lib64/libcudart.so  /home/craig/anaconda3/envs/py/lib/libnvToolsExt.so  /usr/local/cuda/lib64/libcufft.so  /usr/local/cuda/lib64/libcurand.so  /usr/local/cuda/lib64/libcublas.so  /usr/local/cuda/lib64/libcudnn.so  lib/libgtest.a  -pthread &amp;&amp; :
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmACC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTST@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHHH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmSSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBII@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
collect2: error: ld returned 1 exit status
[40/381] Building CXX object test_api/CMakeFiles/test_api.dir/modules.cpp.o
ninja: build stopped: subcommand failed.
</code></pre>

ptrblck: <p>Based on the error message it seems that linking cublas fails.<br>
Did you install different CUDA toolkit versions or tried to manipulate the cublas version manually somehow?</p>

craigyang: <p>Thanks for the instant reply! I am building on a Ubuntu cluster with <code>CUDA_HOME</code> set to the directory of a preinstalled CUDA toolkit. There are indeed other CUDA versions installed as well, but not in that directory (I think, but will double check). Just curious, how will this impact the linking?</p>
<p>For reference, below is what’s in the <code>version.json</code> file in that <code>CUDA_HOME</code> directory:</p>
<pre><code class="lang-auto">{
   "cuda" : {
      "name" : "CUDA SDK",
      "version" : "11.2.20210204"
   },
   "cuda_cudart" : {
      "name" : "CUDA Runtime (cudart)",
      "version" : "11.2.146"
   },
   "cuda_cuobjdump" : {
      "name" : "cuobjdump",
      "version" : "11.2.135"
   },
   "cuda_cupti" : {
      "name" : "CUPTI",
      "version" : "11.2.135"
   },
   "cuda_cuxxfilt" : {
      "name" : "CUDA cu++ filt",
      "version" : "11.2.135"
   },
   "cuda_demo_suite" : {
      "name" : "CUDA Demo Suite",
      "version" : "11.2.67"
   },
   "cuda_gdb" : {
      "name" : "CUDA GDB",
      "version" : "11.2.135"
   },
   "cuda_memcheck" : {
      "name" : "CUDA Memcheck",
      "version" : "11.2.135"
   },
   "cuda_nsight" : {
      "name" : "Nsight Eclipse Plugins",
      "version" : "11.2.135"
   },
   "cuda_nvcc" : {
      "name" : "CUDA NVCC",
      "version" : "11.2.142"
},
   "cuda_nvdisasm" : {
      "name" : "CUDA nvdisasm",
      "version" : "11.2.135"
   },
   "cuda_nvml_dev" : {
      "name" : "CUDA NVML Headers",
      "version" : "11.2.67"
   },
   "cuda_nvprof" : {
      "name" : "CUDA nvprof",
      "version" : "11.2.135"
   },
   "cuda_nvprune" : {
      "name" : "CUDA nvprune",
      "version" : "11.2.135"
   },
   "cuda_nvrtc" : {
      "name" : "CUDA NVRTC",
      "version" : "11.2.142"
   },
   "cuda_nvtx" : {
      "name" : "CUDA NVTX",
      "version" : "11.2.67"
   },
   "cuda_nvvp" : {
      "name" : "CUDA NVVP",
      "version" : "11.2.135"
   },
   "cuda_samples" : {
      "name" : "CUDA Samples",
      "version" : "11.2.135"
   },
   "cuda_sanitizer_api" : {
      "name" : "CUDA Compute Sanitizer API",
      "version" : "11.2.135"
  },
   "libcublas" : {
      "name" : "CUDA cuBLAS",
      "version" : "11.4.1.1026"
   },
   "libcufft" : {
      "name" : "CUDA cuFFT",
      "version" : "10.4.0.135"
   },
   "libcurand" : {
      "name" : "CUDA cuRAND",
      "version" : "10.2.3.135"
   },
   "libcusolver" : {
      "name" : "CUDA cuSOLVER",
      "version" : "11.1.0.135"
   },
   "libcusparse" : {
      "name" : "CUDA cuSPARSE",
      "version" : "11.4.0.135"
   },
   "libnpp" : {
      "name" : "CUDA NPP",
      "version" : "11.3.2.139"
   },
   "libnvjpeg" : {
      "name" : "CUDA nvJPEG",
      "version" : "11.4.0.135"
   },
   "nsight_compute" : {
      "name" : "Nsight Compute",
      "version" : "2020.3.1.3"
   },
   "nsight_systems" : {
      "name" : "Nsight Systems",
      "version" : "2020.4.3.7"
   },
   "nvidia_driver" : {
      "name" : "NVIDIA Linux Driver",
      "version" : "460.32.03"
   }
}
</code></pre>

lcxywfe: <p>I met the same problem, and why cuda toolkit in other directory which not in <code>LIBRARY_PATH</code> could disturb linking?</p>

lamb_ele: <p>I’m getting the same error when I try to install pytorch from the source with cuda 11.4</p>
<pre><code class="lang-auto">[1/871] Building CXX object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp.o
FAILED: caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp.o 
/usr/bin/c++ -DAT_PER_OPERATOR_HEADERS -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DIDEEP_USE_MKL -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DTORCH_CUDA_BUILD_MAIN_LIB -DUSE_C10D_GLOO -DUSE_C10D_MPI -DUSE_C10D_NCCL -DUSE_CUDA -DUSE_DISTRIBUTED -DUSE_EXPERIMENTAL_CUDNN_V8_API -DUSE_EXTERNAL_MZCRC -DUSE_NCCL -DUSE_RPC -DUSE_TENSORPIPE -D_FILE_OFFSET_BITS=64 -Dtorch_cuda_EXPORTS -I/home/jupyter/pytorch/build/aten/src -I/home/jupyter/pytorch/aten/src -I/home/jupyter/pytorch/build -I/home/jupyter/pytorch -I/home/jupyter/pytorch/cmake/../third_party/benchmark/include -I/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include -I/home/jupyter/pytorch/third_party/onnx -I/home/jupyter/pytorch/build/third_party/onnx -I/home/jupyter/pytorch/third_party/foxi -I/home/jupyter/pytorch/build/third_party/foxi -I/home/jupyter/pytorch/build/include -I/home/jupyter/pytorch/torch/csrc/distributed -I/home/jupyter/pytorch/aten/src/THC -I/home/jupyter/pytorch/aten/src/ATen/cuda -I/home/jupyter/pytorch/build/caffe2/aten/src -I/home/jupyter/pytorch/aten/../third_party/catch/single_include -I/home/jupyter/pytorch/aten/src/ATen/.. -I/home/jupyter/pytorch/build/nccl/include -I/home/jupyter/pytorch/c10/cuda/../.. -I/home/jupyter/pytorch/c10/.. -I/home/jupyter/pytorch/third_party/tensorpipe -I/home/jupyter/pytorch/build/third_party/tensorpipe -I/home/jupyter/pytorch/third_party/tensorpipe/third_party/libnop/include -I/home/jupyter/pytorch/torch/csrc/api -I/home/jupyter/pytorch/torch/csrc/api/include -isystem /home/jupyter/pytorch/build/third_party/gloo -isystem /home/jupyter/pytorch/cmake/../third_party/gloo -isystem /home/jupyter/pytorch/cmake/../third_party/googletest/googlemock/include -isystem /home/jupyter/pytorch/cmake/../third_party/googletest/googletest/include -isystem /home/jupyter/pytorch/third_party/protobuf/src -isystem /opt/conda/include -isystem /home/jupyter/pytorch/third_party/gemmlowp -isystem /home/jupyter/pytorch/third_party/neon2sse -isystem /home/jupyter/pytorch/third_party/XNNPACK/include -isystem /home/jupyter/pytorch/third_party/ittapi/include -isystem /home/jupyter/pytorch/cmake/../third_party/eigen -isystem /usr/local/cuda/include -isystem /home/jupyter/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN/include -isystem /home/jupyter/pytorch/third_party/ideep/include -isystem /home/jupyter/pytorch/third_party/ideep/mkl-dnn/include -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -fPIC -DCAFFE2_USE_GLOO -DTH_HAVE_THREAD -Wall -Wextra -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-missing-field-initializers -Wno-write-strings -Wno-unknown-pragmas -Wno-type-limits -Wno-array-bounds -Wno-sign-compare -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-missing-braces -Wno-maybe-uninitialized -fvisibility=hidden -O2 -DTORCH_CUDA_BUILD_MAIN_LIB -pthread -std=gnu++14 -MD -MT caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp.o -MF caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp.o.d -o caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp.o -c /home/jupyter/pytorch/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp
In file included from /home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_ConvDesc.h:35,
                 from /home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend.h:100,
                 from /home/jupyter/pytorch/aten/src/ATen/native/quantized/cudnn/utils.h:20,
                 from /home/jupyter/pytorch/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp:11:
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_utils.h: In function ‘std::__cxx11::string cudnn_frontend::to_string(cudnnDataType_t)’:
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_utils.h:93:14: error: ‘CUDNN_DATA_INT64’ was not declared in this scope
         case CUDNN_DATA_INT64:
              ^~~~~~~~~~~~~~~~
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_utils.h:93:14: note: suggested alternative: ‘CUDNN_DATA_INT8x4’
         case CUDNN_DATA_INT64:
              ^~~~~~~~~~~~~~~~
              CUDNN_DATA_INT8x4
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_utils.h:95:14: error: ‘CUDNN_DATA_BFLOAT16’ was not declared in this scope
         case CUDNN_DATA_BFLOAT16:
              ^~~~~~~~~~~~~~~~~~~
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_utils.h:95:14: note: suggested alternative: ‘CUDNN_DATA_FLOAT’
         case CUDNN_DATA_BFLOAT16:
              ^~~~~~~~~~~~~~~~~~~
              CUDNN_DATA_FLOAT
In file included from /home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_Operation.h:36,
                 from /home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_OperationGraph.h:36,
                 from /home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_Heuristics.h:31,
                 from /home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend.h:101,
                 from /home/jupyter/pytorch/aten/src/ATen/native/quantized/cudnn/utils.h:20,
                 from /home/jupyter/pytorch/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp:11:
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_PointWiseDesc.h: In member function ‘int64_t cudnn_frontend::PointWiseDesc_v8::getPortCount() const’:
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_PointWiseDesc.h:87:18: error: ‘CUDNN_POINTWISE_RELU_BWD’ was not declared in this scope
             case CUDNN_POINTWISE_RELU_BWD:
                  ^~~~~~~~~~~~~~~~~~~~~~~~
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_PointWiseDesc.h:87:18: note: suggested alternative: ‘CUDNN_POINTWISE_RELU_FWD’
             case CUDNN_POINTWISE_RELU_BWD:
                  ^~~~~~~~~~~~~~~~~~~~~~~~
                  CUDNN_POINTWISE_RELU_FWD
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_PointWiseDesc.h:88:18: error: ‘CUDNN_POINTWISE_TANH_BWD’ was not declared in this scope
             case CUDNN_POINTWISE_TANH_BWD:
                  ^~~~~~~~~~~~~~~~~~~~~~~~
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_PointWiseDesc.h:88:18: note: suggested alternative: ‘CUDNN_POINTWISE_TANH_FWD’
             case CUDNN_POINTWISE_TANH_BWD:
....
ninja: build stopped: subcommand failed.
Building wheel torch-1.13.0a0+gitc949788
-- Building version 1.13.0a0+gitc949788
cmake --build . --target install --config Release -- -j 8
</code></pre>

ptrblck: <pre><code class="lang-auto">/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_utils.h:93:14: error: ‘CUDNN_DATA_INT64’ was not declared in this scope
</code></pre>
<p>Did you initialize all submodules before starting the build?</p>

lamb_ele: <p>I was following the instructions from pytorch github readme: <a href="https://github.com/pytorch/pytorch#from-source" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a> Specifically I ran: <code>git submodule update --init --recursive --jobs 0 before running</code>:</p>
<pre><code class="lang-auto">export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py install
</code></pre>
<p>which gave the error I posted above. Now that I saw this other link and ran <code>git submodule update --init --recursive</code> instead and got different errors.</p>
<p>Anyways, I gave up. It shouldn’t be this hard to get pytorch to work with gpu. I’m selecting a different VM instance that has everything set up already. Thanks for the help</p>

ptrblck: <p>Yes, you are right and the common approach is to install the pip wheels or conda binaries which already ship with the CUDA runtime. If you need to latest nightly release, you can also install the nightly binaries with the corresponding CUDA version.<br>
A source build would be needed in case you want a specific CUDA/cuDNN/NCCL etc. version or want to add changes to the framework and while I haven’t seen your issue before (if seems you indeed did not synchronize the submodules) I would also consider it a more complicated approach as you would need a locally installed and working build toolchain (C++ compiler, CUDA compiler etc.).</p>

Thomas_Buddenkotte: <p>Hi all,<br>
a few months later I hit the same problem again. I used the same commands as before (git submodule …) and exported USE_ROCM=0 and CUDNN_{LIB_DIR/INCLUDE_DIR/LIBRARY} to the library and pathes of the cudnn files. I now get the following error:</p>
<p>[5410/6989] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterCPU.cpp.o<br>
[5411/6989] Performing build step for ‘nccl_external’<br>
make -C src build BUILDDIR=/local/scratch/public/tb588/github/pytorch/build/nccl<br>
make[1]: Entering directory ‘/local/scratch/public/tb588/github/pytorch/third_party/nccl/nccl/src’<br>
Grabbing   include/nccl_net.h                  &gt; /local/scratch/public/tb588/github/pytorch/build/nccl/include/nccl_net.h<br>
Generating nccl.h.in                           &gt; /local/scratch/public/tb588/github/pytorch/build/nccl/include/nccl.h<br>
Generating nccl.pc.in                          &gt; /local/scratch/public/tb588/github/pytorch/build/nccl/lib/pkgconfig/nccl.pc<br>
Compiling  init.cc                             &gt; /local/scratch/public/tb588/github/pytorch/build/nccl/obj/init.o<br>
Compiling  channel.cc                          &gt; /local/scratch/public/tb588/github/pytorch/build/nccl/obj/channel.o<br>
Compiling  bootstrap.cc                        &gt; /local/scratch/public/tb588/github/pytorch/build/nccl/obj/bootstrap.o<br>
Compiling  transport.cc                        &gt; /local/scratch/public/tb588/github/pytorch/build/nccl/obj/transport.o<br>
Compiling  enqueue.cc                          &gt; /local/scratch/public/tb588/github/pytorch/build/nccl/obj/enqueue.o<br>
enqueue.cc: In function ‘ncclResult_t scheduleCollTasksToPlan(ncclComm*, ncclKernelPlan*, int*)’:<br>
enqueue.cc:578:60: warning: ‘workFuncIndex’ may be used uninitialized in this function [-Wmaybe-uninitialized]<br>
plan-&gt;kernelSpecialized = ncclKerns[workFuncIndex].specialized;<br>
~~~~~~~~~~~~~~~




^




<br>
Compiling  group.cc                            &gt; /local/scratch/public/tb588/github/pytorch/build/nccl/obj/group.o<br>
…<br>
Compiling  functions.cu                        &gt; /local/scratch/public/tb588/github/pytorch/build/nccl/obj/collectives/device/functions.o<br>
Compiling  onerank_reduce.cu                   &gt; /local/scratch/public/tb588/github/pytorch/build/nccl/obj/collectives/device/onerank_reduce.o<br>
Archiving  objects                             &gt; /local/scratch/public/tb588/github/pytorch/build/nccl/obj/collectives/device/colldevice.a<br>
make[2]: Leaving directory ‘/local/scratch/public/tb588/github/pytorch/third_party/nccl/nccl/src/collectives/device’<br>
Linking    libnccl.so.2.14.3                   &gt; /local/scratch/public/tb588/github/pytorch/build/nccl/lib/libnccl.so.2.14.3<br>
Archiving  libnccl_static.a                    &gt; /local/scratch/public/tb588/github/pytorch/build/nccl/lib/libnccl_static.a<br>
make[1]: Leaving directory ‘/local/scratch/public/tb588/github/pytorch/third_party/nccl/nccl/src’<br>
ninja: build stopped: subcommand failed.<br>
Building wheel torch-2.0.0a0+gitf40183d<br>
– Building version 2.0.0a0+gitf40183d<br>
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/local/scratch/public/tb588/github/pytorch/torch -DCMAKE_PREFIX_PATH=/local/scratch/public/tb588/miniconda3/e<br>
nvs/pytorch_env/lib/python3.8/site-packages;/local/scratch/public/tb588/miniconda3/envs/pytorch_env -DCMAKE_PREFIX_PATH_modshare=/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/cuda-11.1.1-joqpcn7aufehhcop2<br>
vxsczwivpgsgi4o/:1:/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gmp-6.1.2-qc4qcfz4monpllc3nqupdo7vwinf73sw/:1:/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/cudnn-8.0.5.39-11.1-aogmnpiibuxgcu47oecr<br>
ba66pm44a5yd/:1:/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpc-1.0.3-tumbpshu5hjxwextoudk5hmic6nspb3z/:1:/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/cuda-11.2.0-4cnusrbsscayn2fudsrlkkwmxuatqnn<br>
q/:1:/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/isl-0.18-vttqoutnsmjpm3ogb52rninksc7hq5ax/:1:/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpfr-3.1.5-mdi6irzvxcbemt7yredzr36dvo6ty4sl/:1:/alt/app<br>
lic/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-gmnpoqci26qz55qnegn2kvt7hq67hnsy/:1 -DCUDNN_INCLUDE_DIR=/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/cudnn-8.0.5.39-11.1-aogmnpiibuxgcu47oecrba66pm4<br>
4a5yd/include -DCUDNN_LIBRARY=/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/cudnn-8.0.5.39-11.1-aogmnpiibuxgcu47oecrba66pm44a5yd/lib64 -DNUMPY_INCLUDE_DIR=/local/scratch/public/tb588/miniconda3/envs/pytor<br>
ch_env/lib/python3.8/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/local/scratch/public/tb588/miniconda3/envs/pytorch_env/bin/python -DPYTHON_INCLUDE_DIR=/local/scratch/public/tb588/miniconda3/envs/pytor<br>
ch_env/include/python3.8 -DPYTHON_LIBRARY=/local/scratch/public/tb588/miniconda3/envs/pytorch_env/lib/libpython3.8.so.1.0 -DTORCH_BUILD_VERSION=2.0.0a0+gitf40183d -DUSE_NUMPY=True -DUSE_ROCM=0 /local/scratch/pub<br>
lic/tb588/github/pytorch<br>
cmake --build . --target install --config Release</p>
<p>Does anyone know where this error might come from?</p>
<p>Many thanks!</p>