Fattal error: torch/extension.h: No such file or directory

Ron: <pre><code class="lang-auto">OS: Ubuntu 22.04 LTS
Python: 3.10.6
gcc: 1.13.0
ninja: 1.8.2
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1893, in _run_ninja_build
    subprocess.run(
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ron/.vscode/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in &lt;module&gt;
    cli.main()
  File "/home/ron/.vscode/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/home/ron/.vscode/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/home/ron/.vscode/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/home/ron/.vscode/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/home/ron/.vscode/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/sample/generate.py", line 22, in &lt;module&gt;
    import test_ddgan as dg
  File "/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/./test_ddgan.py", line 14, in &lt;module&gt;
    from score_sde.models.ncsnpp_generator_adagn import NCSNpp
  File "/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/./score_sde/models/ncsnpp_generator_adagn.py", line 34, in &lt;module&gt;
    from . import utils, layers, layerspp, dense_layer
  File "/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/./score_sde/models/layerspp.py", line 33, in &lt;module&gt;
    from . import up_or_down_sampling, dense_layer
  File "/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/./score_sde/models/up_or_down_sampling.py", line 15, in &lt;module&gt;
    from score_sde.op import upfirdn2d
  File "/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/./score_sde/op/__init__.py", line 1, in &lt;module&gt;
    from .fused_act import FusedLeakyReLU, fused_leaky_relu
  File "/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/./score_sde/op/fused_act.py", line 19, in &lt;module&gt;
    fused = load(
  File "/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1284, in load
    return _jit_compile(
  File "/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1509, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1624, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1909, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'fused': [1/3] /usr/local/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include/TH -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++17 -c '/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/score_sde/op/fused_bias_act_kernel.cu' -o fused_bias_act_kernel.cuda.o 
FAILED: fused_bias_act_kernel.cuda.o 
/usr/local/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include/TH -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++17 -c '/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/score_sde/op/fused_bias_act_kernel.cu' -o fused_bias_act_kernel.cuda.o 
nvcc fatal   : A single input file is required for a non-link phase when an outputfile is specified
[2/3] c++ -MMD -MF fused_bias_act.o.d -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include/TH -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -c '/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/score_sde/op/fused_bias_act.cpp' -o fused_bias_act.o 
FAILED: fused_bias_act.o 
c++ -MMD -MF fused_bias_act.o.d -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include/TH -isystem /home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/mdmgan/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -c '/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/score_sde/op/fused_bias_act.cpp' -o fused_bias_act.o 
/home/ron/Classes/Adv Computer Vision/Proj/diffgan test/denoising-diffusion-gan/score_sde/op/fused_bias_act.cpp:8:10: fatal error: torch/extension.h: No such file or directory
    8 | #include &lt;torch/extension.h&gt;
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
</code></pre>
<pre><code class="lang-auto">Package                  Version
------------------------ ----------
absl-py                  1.4.0
anyio                    3.6.2
argon2-cffi              21.3.0
argon2-cffi-bindings     21.2.0
arrow                    1.2.3
asttokens                2.2.1
attrs                    22.2.0
backcall                 0.2.0
beautifulsoup4           4.11.2
bleach                   6.0.0
blis                     0.7.9
cachetools               5.3.0
catalogue                2.0.8
certifi                  2022.12.7
cffi                     1.15.1
charset-normalizer       3.1.0
chumpy                   0.70
click                    8.1.3
cmake                    3.26.0
comm                     0.1.2
confection               0.0.4
contourpy                1.0.7
cycler                   0.11.0
cymem                    2.0.7
debugpy                  1.6.6
decorator                5.1.1
defusedxml               0.7.1
executing                1.2.0
fastjsonschema           2.16.3
filelock                 3.10.0
fonttools                4.39.2
fqdn                     1.5.1
google-auth              2.16.2
google-auth-oauthlib     0.4.6
grpcio                   1.51.3
idna                     3.4
ipykernel                6.21.3
ipython                  8.11.0
ipython-genutils         0.2.0
ipywidgets               8.0.4
isoduration              20.11.0
jedi                     0.18.2
Jinja2                   3.1.2
jsonpointer              2.3
jsonschema               4.17.3
jupyter                  1.0.0
jupyter_client           8.0.3
jupyter-console          6.6.3
jupyter_core             5.3.0
jupyter-events           0.6.3
jupyter_server           2.5.0
jupyter_server_terminals 0.4.4
jupyterlab-pygments      0.2.2
jupyterlab-widgets       3.0.5
kiwisolver               1.4.4
langcodes                3.3.0
lit                      15.0.7
lmdb                     1.4.0
Markdown                 3.4.1
MarkupSafe               2.1.2
matplotlib               3.7.1
matplotlib-inline        0.1.6
mistune                  2.0.5
mpmath                   1.3.0
murmurhash               1.0.9
nbclassic                0.5.3
nbclient                 0.7.2
nbconvert                7.2.10
nbformat                 5.7.3
nest-asyncio             1.5.6
networkx                 3.0
ninja                    1.11.1
notebook                 6.5.3
notebook_shim            0.2.2
numpy                    1.23.1
nvidia-cublas-cu11       11.10.3.66
nvidia-cuda-cupti-cu11   11.7.101
nvidia-cuda-nvrtc-cu11   11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11        8.5.0.96
nvidia-cufft-cu11        10.9.0.58
nvidia-curand-cu11       10.2.10.91
nvidia-cusolver-cu11     11.4.0.1
nvidia-cusparse-cu11     11.7.4.91
nvidia-nccl-cu11         2.14.3
nvidia-nvtx-cu11         11.7.91
oauthlib                 3.2.2
packaging                23.0
pandocfilters            1.5.0
parso                    0.8.3
pathy                    0.10.1
pexpect                  4.8.0
pickleshare              0.7.5
Pillow                   9.4.0
pip                      22.0.2
platformdirs             3.1.1
preshed                  3.0.8
prometheus-client        0.16.0
prompt-toolkit           3.0.38
protobuf                 3.20.3
psutil                   5.9.4
ptyprocess               0.7.0
pure-eval                0.2.2
pyasn1                   0.4.8
pyasn1-modules           0.2.8
pycparser                2.21
pydantic                 1.10.6
Pygments                 2.14.0
pyparsing                3.0.9
pyrsistent               0.19.3
python-dateutil          2.8.2
python-json-logger       2.0.7
PyYAML                   6.0
pyzmq                    25.0.1
qtconsole                5.4.1
QtPy                     2.3.0
requests                 2.28.2
requests-oauthlib        1.3.1
rfc3339-validator        0.1.4
rfc3986-validator        0.1.1
rsa                      4.9
scipy                    1.10.1
Send2Trash               1.8.0
setuptools               59.6.0
six                      1.16.0
smart-open               6.3.0
smplx                    0.1.28
sniffio                  1.3.0
soupsieve                2.4
spacy                    3.5.1
spacy-legacy             3.0.12
spacy-loggers            1.0.4
srsly                    2.4.6
stack-data               0.6.2
sympy                    1.11.1
tensorboard              2.12.0
tensorboard-data-server  0.7.0
tensorboard-plugin-wit   1.8.1
tensorboardX             2.6
terminado                0.17.1
thinc                    8.1.9
tinycss2                 1.2.1
torch                    2.0.0
torchvision              0.15.1
tornado                  6.2
tqdm                     4.65.0
traitlets                5.9.0
triton                   2.0.0
typer                    0.7.0
typing_extensions        4.5.0
uri-template             1.2.0
urllib3                  1.26.15
wasabi                   1.1.1
wcwidth                  0.2.6
webcolors                1.12
webencodings             0.5.1
websocket-client         1.5.1
Werkzeug                 2.2.3
wheel                    0.40.0
widgetsnbextension       4.0.5
</code></pre>
<p>nvidia-smi<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26ad222f9ceaab31988d512acf226ef3ab53322e.png" data-download-href="https://discuss.pytorch.org/uploads/default/26ad222f9ceaab31988d512acf226ef3ab53322e" title="Screenshot from 2023-03-16 23-40-53"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/26ad222f9ceaab31988d512acf226ef3ab53322e_2_690x326.png" alt="Screenshot from 2023-03-16 23-40-53" data-base62-sha1="5w950uCMTzjGQAwLlpksFPuv6Gy" width="690" height="326" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/26ad222f9ceaab31988d512acf226ef3ab53322e_2_690x326.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/26ad222f9ceaab31988d512acf226ef3ab53322e_2_1035x489.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/6/26ad222f9ceaab31988d512acf226ef3ab53322e_2_1380x652.png 2x" data-dominant-color="3D1931"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-03-16 23-40-53</span><span class="informations">1608×762 97.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
nvcc --version<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/61ef46393153975fa839fe7c024e5c856dc16652.png" data-download-href="https://discuss.pytorch.org/uploads/default/61ef46393153975fa839fe7c024e5c856dc16652" title="Screenshot from 2023-03-16 23-41-44"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61ef46393153975fa839fe7c024e5c856dc16652_2_690x158.png" alt="Screenshot from 2023-03-16 23-41-44" data-base62-sha1="dYmXmy1kJduQjjAoqq3CmnsIWyK" width="690" height="158" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61ef46393153975fa839fe7c024e5c856dc16652_2_690x158.png, https://discuss.pytorch.org/uploads/default/original/3X/6/1/61ef46393153975fa839fe7c024e5c856dc16652.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/1/61ef46393153975fa839fe7c024e5c856dc16652.png 2x" data-dominant-color="4A293F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-03-16 23-41-44</span><span class="informations">844×194 38.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Question</strong></p>
<p>How can I resolve this issue?</p>
<p>My cuda-toolkit version is the latest and compatible with my driver, so that isn’t the issue.  The nvcc path is set correctly, and I have also set <code>CUDA_HOME=/urs/local/cuda</code>.  There are plenty of similar problems out there, and all of these were suggested as solutions, but haven’t actually fixed my problem.  Any guidance is much appreciated.</p>

ptrblck: <p>I don’t know why your environment isn’t able to detect the needed <code>torch/extension.h</code> file as it’s located in <code>.../site-packages/torch/include/torch/extension.h</code> in the current <code>2.0.0</code> release.<br>
I also don’t know what exactly you are running to reproduce the issue, but executing <code>python fused_act.py</code> in the linked repository seems to build something without a failure in my env.<br>
With that being said, it seems the repository depends on <code>torch==1.8.0</code> as seen <a href="https://github.com/NVlabs/denoising-diffusion-gan/blob/6818ded6443e10ab69c7864745457ce391d4d883/requirements.txt#L1">here</a>, so I’m unsure if my build really succeeded as I don’t know how to reproduce your error.</p>