Ninja: build stopped: subcommand failed

octadion: <p>my gcc version is 9.4.0</p>
<p>Traceback (most recent call last):<br>
File “/opt/conda/envs/difgan/lib/python3.9/site-packages/torch/utils/cpp_extension.py”, line 1808, in _run_ninja_build<br>
subprocess.run(<br>
File “/opt/conda/envs/difgan/lib/python3.9/subprocess.py”, line 528, in run<br>
raise CalledProcessError(retcode, process.args,<br>
subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’]’ returned non-zero exit status 1.</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>Traceback (most recent call last):<br>
File “/home/octadion/diffusion-gan/Diffusion-GAN/diffusion-stylegan2/torch_utils/ops/bias_act.py”, line 41, in <em>init<br>
<em>plugin = custom_ops.get_plugin(‘bias_act_plugin’, sources=sources, extra_cuda_cflags=[‘–use_fast_math’])<br>
File “/home/octadion/diffusion-gan/Diffusion-GAN/diffusion-stylegan2/torch_utils/custom_ops.py”, line 103, in get_plugin<br>
torch.utils.cpp_extension.load(name=module_name, verbose=verbose_build, sources=sources, **build_kwargs)<br>
File “/opt/conda/envs/difgan/lib/python3.9/site-packages/torch/utils/cpp_extension.py”, line 1202, in load<br>
return <em>jit_compile(<br>
File “/opt/conda/envs/difgan/lib/python3.9/site-packages/torch/utils/cpp_extension.py”, line 1425, in <em>jit_compile<br>
<em>write_ninja_file_and_build_library(<br>
File “/opt/conda/envs/difgan/lib/python3.9/site-packages/torch/utils/cpp_extension.py”, line 1537, in <em>write_ninja_file_and_build_library<br>
<em>run_ninja_build(<br>
File “/opt/conda/envs/difgan/lib/python3.9/site-packages/torch/utils/cpp_extension.py”, line 1824, in <em>run_ninja_build<br>
raise RuntimeError(message) from e<br>
RuntimeError: Error building extension ‘bias_act_plugin’: [1/3] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=“<em>gcc" -DPYBIND11_STDLIB="<em>libstdcpp" -DPYBIND11_BUILD_ABI="<em>cxxabi1011" -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/envs/difgan/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS</em></em> -D__CUDA_NO_HALF_CONVERSIONS</em></em> -D__CUDA_NO_BFLOAT16_CONVERSIONS</em></em> -D__CUDA_NO_HALF2_OPERATORS</em></em> --expt-relaxed-constexpr -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 --compiler-options ‘-fPIC’ --use_fast_math -std=c++14 -c /home/octadion/diffusion-gan/Diffusion-GAN/diffusion-stylegan2/torch_utils/ops/bias_act.cu -o bias_act.cuda.o<br>
FAILED: bias_act.cuda.o<br>
/usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=“<em>gcc" -DPYBIND11_STDLIB="<em>libstdcpp" -DPYBIND11_BUILD_ABI="<em>cxxabi1011" -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/envs/difgan/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS</em></em> -D__CUDA_NO_HALF_CONVERSIONS</em></em> -D__CUDA_NO_BFLOAT16_CONVERSIONS</em></em> -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 --compiler-options ‘-fPIC’ --use_fast_math -std=c++14 -c /home/octadion/diffusion-gan/Diffusion-GAN/diffusion-stylegan2/torch_utils/ops/bias_act.cu -o bias_act.cuda.o<br>
/bin/sh: 1: /usr/local/cuda/bin/nvcc: not found<br>
[2/3] c++ -MMD -MF bias_act.o.d -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=”_gcc” -DPYBIND11_STDLIB=“_libstdcpp” -DPYBIND11_BUILD_ABI=“_cxxabi1011” -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/envs/difgan/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/octadion/diffusion-gan/Diffusion-GAN/diffusion-stylegan2/torch_utils/ops/bias_act.cpp -o bias_act.o<br>
FAILED: bias_act.o<br>
c++ -MMD -MF bias_act.o.d -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=“_gcc” -DPYBIND11_STDLIB=“_libstdcpp” -DPYBIND11_BUILD_ABI=“_cxxabi1011” -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/envs/difgan/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/octadion/diffusion-gan/Diffusion-GAN/diffusion-stylegan2/torch_utils/ops/bias_act.cpp -o bias_act.o<br>
In file included from /home/octadion/diffusion-gan/Diffusion-GAN/diffusion-stylegan2/torch_utils/ops/bias_act.cpp:10:<br>
/opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/ATen/cuda/CUDAContext.h:5:10: fatal error: cuda_runtime_api.h: No such file or directory<br>
5 | <span class="hashtag">#include</span> &lt;cuda_runtime_api.h&gt;<br>
| ^~~~~~~~~~~~~~~~~~~~<br>
compilation terminated.<br>
ninja: build stopped: subcommand failed.</p>

ptrblck: <p>You didn’t explain your use case, but just posted a stacktrace, so I don’t know if you are trying to build PyTorch from source or a custom extension.<br>
In any case, the error:</p>
<pre><code class="lang-python">/opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/ATen/cuda/CUDAContext.h:5:10: fatal error: cuda_runtime_api.h: No such file or directory
</code></pre>
<p>points to a missing file and it seems your locally installed CUDA toolkit might be missing these files, so make sure this file is available in your CUDA toolkit e.g. at:</p>
<pre><code class="lang-python">/usr/local/cuda-11.7/targets/x86_64-linux/include/cuda_runtime_api.h
</code></pre>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>

tom: <p>Apparently, your installation are missing <code>cuda_runtime_api.h</code>, typically found in <code>/usr/local/cuda/include</code> or so. Maybe the CUDA installation is not quite complete?</p>

octadion: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="176511">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>/usr/local/cuda</code></p>
</blockquote>
</aside>
<p>i tried to run diffusion-style gan sir i installed pytorch with this <code>pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl /torch_stable.html</code> and yes it turns out that after I checked the runtime_api was missing, how could this happen even though the download and install process had finished to the end? and there’s 2 cuda version in <code>/usr/local</code>, i think because previously i tried to install 2 different version of cuda. both cuda have no runtime_api, should i uninstall one of them?</p>

octadion: <p>yes sir it turns out that after I checked the runtime_api was missing, how could this happen even though the download and install process had finished to the end?</p>

ptrblck: <p>PyTorch itself won’t need this file if you execute a workload using an installed binary unless you are trying to build a custom extension. Is this the case and if so, what are you trying to build? You can use a <code>verbose=True</code>  in the extension builds for more verbose outputs during the build.</p>

octadion: <p>actually i tried to run stylegan2-ada-pytorch / diffusion-stylegan2 sir. on my different machines, everything works normally.</p>