PyTorch can't find CUDA header cuda_runtime_api.h

GriffinBonner: <p>I am developing a dynamic c++ library that uses the PyTorch c++ front-end internally and <em>cannot</em> impose an external dependence on PyTorch shared libraries <em>(libtorch.so, libtorch_cuda.so, libtorch_cpu.so)</em> on end-users. It is based on an existing c++ application that dynamically links PyTorch shared libraries. Thus, the library needs to link the static versions of PyTorch <em>(libtorch.a, libtorch_cuda.a, libtorch_cpu.a)</em>.</p>
<p><strong>CMake</strong></p>
<ul>
<li>I locate Torch and CUDA-Toolkit in CMake as follows</li>
</ul>
<pre><code class="lang-auto">    find_package(CUDAToolkit 11.8.0 EXACT)
    find_package(Torch)
</code></pre>
<p><strong>Issues</strong></p>
<p><strong>1. PyTorch cannot find CUDA header: cuda_runtime_api.h</strong></p>
<ul>
<li>Compiling the target <em>(my library)</em> that links PyTorch static libraries produces this error:</li>
</ul>
<pre><code class="lang-auto">    /opt/deepsig/.venv/lib/python3.8/site-packages/torch/include/c10/cuda/CUDAStream.h:6:10: fatal error: cuda_runtime_api.h: No such file or directory
    6 | #include &lt;cuda_runtime_api.h&gt;
</code></pre>
<ul>
<li>The CUDA-Toolkit is installed in the following location <code>/usr/local/cuda-11.8</code></li>
<li>The following symbolic links also exist:</li>
</ul>
<pre><code class="lang-auto">    ls -ln /usr/local/cuda
    lrwxrwxrwx 1 0 0 22 Jun 20 20:40 /usr/local/cuda -&gt; /etc/alternatives/cuda

    ls -ln /usr/local/cuda-11
    lrwxrwxrwx 1 0 0 25 Jun 20 20:40 /usr/local/cuda-11 -&gt; /etc/alternatives/cuda-11
</code></pre>
<ul>
<li>The file <code>cuda_runtime_api.h</code> exists in the expected location:</li>
</ul>
<pre><code class="lang-auto">    find . -name "cuda_runtime_api.h"
    ./local/cuda-11.8/targets/x86_64-linux/include/cuda_runtime_api.h
</code></pre>
<p><strong>2. find_package(Torch): Torch-Config.cmake cannot find certain <em>(3rd party)</em> libraries. Produces a warning during cmake configure/generate steps.</strong></p>
<pre><code class="lang-auto">    CMake Warning at /opt/deepsig/.venv/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
      static library onnx_LIBRARY-NOTFOUND not found.
    Call Stack (most recent call first):
      /opt/deepsig/.venv/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:104 (append_torchlib_if_found)
      CMakeLists.txt:227 (find_package)

    CMake Warning at /opt/deepsig/.venv/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
      static library foxi_loader_LIBRARY-NOTFOUND not found.
    Call Stack (most recent call first):
      /opt/deepsig/.venv/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:106    (append_torchlib_if_found)
      CMakeLists.txt:227 (find_package)

    CMake Warning at /opt/deepsig/.venv/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
      static library fmt_LIBRARY-NOTFOUND not found.
    Call Stack (most recent call first):
      /opt/deepsig/.venv/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:106 (append_torchlib_if_found)
      CMakeLists.txt:227 (find_package)

    CMake Warning at /opt/deepsig/.venv/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
      static library eigen_blas_LIBRARY-NOTFOUND not found.
    Call Stack (most recent call first):
      /opt/deepsig/.venv/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:113 (append_torchlib_if_found)
      CMakeLists.txt:227 (find_package)
    -- Found Torch: /opt/deepsig/.venv/lib/python3.8/site-packages/torch/lib/libtorch.a
</code></pre>
<p><strong>PyTorch Build Environment</strong><br>
Docker base image: nvidia/cuda:11.8.0-cudnn8-devel-ubuntu20.04<br>
OS: Ubuntu 20.04<br>
Python: 3.8<br>
CUDA: 11.8.0<br>
PyTorch: 1.11.0</p>
<p>I configure the PyTorch build in this docker, build the wheel, and install it in a development docker container used to build my c++ library.</p>
<ol>
<li>
<p><strong>Build PyTorch with static libraries</strong></p>
<p><strong>a. Download CUDA dependencies</strong></p>
<ul>
<li>libnvinfer8=8.5.3-1+cuda11.8</li>
<li>libnvinfer-plugin8=8.5.3-1+cuda11.8</li>
<li>python3-libnvinfer-dev=8.5.3-1+cuda11.8</li>
<li>python3-libnvinfer=8.5.3-1+cuda11.8</li>
<li>libnvinfer-dev=8.5.3-1+cuda11.8</li>
<li>libnvinfer-plugin-dev=8.5.3-1+cuda11.8</li>
<li>libnvparsers-dev=8.5.3-1+cuda11.8</li>
<li>libnvonnxparsers-dev=8.5.3-1+cuda11.8</li>
<li>libnvparsers8=8.5.3-1+cuda11.8</li>
<li>libnvonnxparsers8=8.5.3-1+cuda11.8</li>
</ul>
<p><strong>b. Download PyTorch Library Dependencies</strong></p>
<ul>
<li>git</li>
<li>gnupg</li>
<li>libprotobuf-dev</li>
<li>wget</li>
</ul>
<p><strong>c. Clone PyTorch</strong></p>
<ul>
<li>I have been unable to configure the PyTorch build so that static libraries are included in the wheel package without manually editing <code>setup.py</code> and adding all <code>lib/*.a</code> to <code>torch_package_data</code>.</li>
</ul>
<pre><code class="lang-auto">    git clone https://github.com/pytorch/pytorch.git &amp;&amp; \
    cd pytorch &amp;&amp; \
    git checkout v1.11.0 &amp;&amp; git submodule update --init --recursive &amp;&amp; \
    sed -i '1061s/.*/"lib\/*.a"],/' setup.py
</code></pre>
<p><strong>d. Configure PyTorch and build wheel</strong></p>
<pre><code class="lang-auto">    export BUILD_SHARED_LIBS=0
    export USE_STATIC_MKL=1
    export TORCH_BUILD_TEST=0
    export TORCH_USE_MKLDNN=0
    export TORCH_USE_TENSORPIPE=0
    export USE_KENETO=0
    export CUDA_HOME=/usr/local/cuda-11.8
    export CUDA_TOOLKIT_ROOT_DIR=$CUDA_HOME
    export LIBRARY_PATH=$CUDA_HOME/lib64:$LIBRARY_PATH
    export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
    export CFLAGS="-I$CUDA_HOME/include $CFLAGS"
    cd pytorch
    python setup.py bdist_wheel
</code></pre>
<p><strong>e. Install wheel in development container</strong></p>
<pre><code class="lang-auto">   python -m pip install /path/to/&lt;pytorch&gt;.whl
</code></pre>
</li>
</ol>
<p><strong>2. Result</strong><br>
The PyTorch build process described above produces the following libraries in my development container:</p>
<ul>
<li>libXNNPACK.a, libc10.a, libcaffe2_nvrtc.so, libclog.a, libfbgemm.a, libgloo_cuda.a, libnnpack.a, libprotobuf.a, libpthreadpool.a, libqnnpack.a, libsleef.a, libtorch_cpu.a, libtorch_python.so, libasmjit.a, libc10_cuda.a, libcaffe2_protos.a, libcpuinfo.a, libgloo.a libkineto.a, libprotobuf-lite.a, libprotoc.a, libpytorch_qnnpack.a, libshm.so, libtorch.a, libtorch_cuda.a</li>
</ul>

malfet: <p>libtorch_static + cuda is essentially a no-go as it quickly runs into 2Gb .cubin problem and libcudnn.a performance is much worse that that of a shared library, see <a href="https://github.com/pytorch/pytorch/issues/50153" class="inline-onebox" rel="noopener nofollow ugc">libtorch_cuda.so is missing fast kernels from libcudnn_static.a, therefore statically linked cuDNN could be much slower than dynamically linked · Issue #50153 · pytorch/pytorch · GitHub</a></p>

ptrblck: <p>I don’t think the performance issue is still a thing 3 years later, but of course the size issue could even be worse.</p>