Custom C++ and CUDA Extensions: compiling exception

username: o.o
cooked: <p>When I compile the customized pytorch c++ extension and CUDA kernel with Nijia, the following exception occurs. I don’t know how to solve it. The environment I use is:<br>
ubuntu18.04<br>
python3.8<br>
cuda 11.1<br>
pytorch 1.10.1</p>
<blockquote>
<p>Blockquote<br>
Traceback (most recent call last):<br>
File “/home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py”, line 1717, in _run_ninja_build<br>
subprocess.run(<br>
File “/home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/subprocess.py”, line 516, in run<br>
raise CalledProcessError(retcode, process.args,<br>
subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’]’ returned non-zero exit status 1.</p>
</blockquote>
<blockquote>
<p>Blockquote<br>
The above exception was the direct cause of the following exception:<br>
Traceback (most recent call last):<br>
File “natten/gradcheck.py”, line 11, in <br>
from nattencuda import NATTENAVFunction, NATTENQKRPBFunction<br>
File “/home/vipuser/Desktop/NAT/Neighborhood-Attention-Transformer-main/natten/nattencuda.py”, line 20, in <br>
nattenav_cuda = load(<br>
File “/home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py”, line 1124, in load<br>
return <em>jit_compile(<br>
File “/home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py”, line 1337, in <em>jit_compile<br>
<em>write_ninja_file_and_build_library(<br>
File “/home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py”, line 1449, in <em>write_ninja_file_and_build_library<br>
<em>run_ninja_build(<br>
File “/home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py”, line 1733, in <em>run_ninja_build<br>
raise RuntimeError(message) from e<br>
RuntimeError: Error building extension ‘nattenav_cuda’: [1/3] :/usr/local/cuda-11.1/bin/nvcc  -DTORCH_EXTENSION_NAME=nattenav_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="<em>gcc" -DPYBIND11_STDLIB="<em>libstdcpp" -DPYBIND11_BUILD_ABI="<em>cxxabi1011" -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/TH -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/THC -isystem :/usr/local/cuda-11.1/include -isystem /home/vipuser/anaconda3/envs/venv3.8/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS</em></em> -D__CUDA_NO_HALF_CONVERSIONS</em></em> -D__CUDA_NO_BFLOAT16_CONVERSIONS</em></em> -D__CUDA_NO_HALF2_OPERATORS</em></em> --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options ‘-fPIC’ -std=c++14 -c /home/vipuser/Desktop/NAT/Neighborhood-Attention-Transformer-main/natten/src/nattenav_cuda_kernel.cu -o nattenav_cuda_kernel.cuda.o<br>
FAILED: nattenav_cuda_kernel.cuda.o<br>
:/usr/local/cuda-11.1/bin/nvcc  -DTORCH_EXTENSION_NAME=nattenav_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="<em>gcc" -DPYBIND11_STDLIB="<em>libstdcpp" -DPYBIND11_BUILD_ABI="<em>cxxabi1011" -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/TH -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/THC -isystem :/usr/local/cuda-11.1/include -isystem /home/vipuser/anaconda3/envs/venv3.8/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS</em></em> -D__CUDA_NO_HALF_CONVERSIONS</em></em> -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options ‘-fPIC’ -std=c++14 -c /home/vipuser/Desktop/NAT/Neighborhood-Attention-Transformer-main/natten/src/nattenav_cuda_kernel.cu -o nattenav_cuda_kernel.cuda.o<br>
/bin/sh: 1: :/usr/local/cuda-11.1/bin/nvcc: not found<br>
[2/3] c++ -MMD -MF nattenav_cuda.o.d -DTORCH_EXTENSION_NAME=nattenav_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/TH -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/THC -isystem :/usr/local/cuda-11.1/include -isystem /home/vipuser/anaconda3/envs/venv3.8/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/vipuser/Desktop/NAT/Neighborhood-Attention-Transformer-main/natten/src/nattenav_cuda.cpp -o nattenav_cuda.o<br>
ninja: build stopped: subcommand failed.</p>
</blockquote>

username: ptrblck
cooked: <p>It seems the CUDA compiler (<code>nvcc</code>) cannot be found:</p>
<pre><code class="lang-python">/bin/sh: 1: :/usr/local/cuda-11.1/bin/nvcc: not found
</code></pre>
<p>What does <code>which nvcc</code> return and are you able to find the binary in <code>/usr/local/cuda-11.1/bin</code>?</p>

username: o.o
cooked: <p>thank you for you replay, and the</p>
<blockquote>
<p>which nvcc</p>
</blockquote>
<p>return</p>
<blockquote>
<p>/usr/local/cuda-11.1/bin/nvcc</p>
</blockquote>
<p>And I can find the binary file nvcc in</p>
<blockquote>
<p>/usr/local/cuda-11.1/bin<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48b9a2f1fdc27ed6af1492f967e3d6497e67c586.png" data-download-href="https://discuss.pytorch.org/uploads/default/48b9a2f1fdc27ed6af1492f967e3d6497e67c586" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48b9a2f1fdc27ed6af1492f967e3d6497e67c586.png" alt="image" data-base62-sha1="anm6l32hywDLyPz4fOBKNSoUah0" width="690" height="421" data-dominant-color="D3D0D0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">983×601 14.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</blockquote>
<p>I’m not sure whether the result is expected, please give me more advice. Thanks a lot!</p>

username: ptrblck
cooked: <p>I don’t know what’s causing the issue, but maybe the Python process doesn’t have access to execute <code>nvcc</code> in this folder? Could you check for permission issues and make sure Python is able to compile the code?</p>