Distribution of pytorch extension

GuillaumeLeclerc: <p>Hello,</p>
<p>I implemented a new operator in cuda and would like to make it available it on pip/conda. What is the best way to do it?</p>
<ul>
<li>Ideally I would build a wheel but I’ve met a lot of issues generating something that is remotely portable from an environment/machine to another. Another issue I don’t know how to solve is how to automatically pick the wheel based on the pytorch version the user has.</li>
<li>If users compile when the package is installed then they need <code>nvcc</code> which is not installed with pytorch using the recommended installation commands. Moreover it seems that the GCC version that ships with conda isn’t compatible with cuda 11.7 so it seems I would need users to have a custom combination of GCC/cuda installed outside of conda.</li>
</ul>
<p>Am I missing something? What is the recommended way of distributing pytorch extensions ?</p>
<p>Thanks for the help</p>

GuillaumeLeclerc: <p>Related to the wheel for example a wheel compiled on another machine with the same version of python, the same major version of pytorch yield to this error when imported:</p>
<pre><code class="lang-auto">site-packages/fast_jl.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC2ENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
</code></pre>
<p>(which is a symbol related to pytorch)</p>

ptrblck: <p>Creating binaries is not trivial if you are shipping C++/CUDA code and you could take a look at our <a href="https://github.com/pytorch/builder"><code>pytorch/builder</code></a> repository to see how the general process is done for PyTorch.<br>
Ideally, you would ship your package with a statically linked CUDA runtime library and would then link against PyTorch and its dependencies.</p>

GuillaumeLeclerc: <p>So statically link against CUDA and dynamic link against pytorch ? What if pytorch upgrades ?</p>
<p>Do you have an example of such thing ?</p>

GuillaumeLeclerc: <p>I can’t even have it compile reliably at install time with GCC 11.3 and the latest version of pytorch it fails with this issue:</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/NVlabs/instant-ngp/issues/119">
  <header class="source">

      <a href="https://github.com/NVlabs/instant-ngp/issues/119" target="_blank" rel="noopener nofollow ugc">github.com/NVlabs/instant-ngp</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/NVlabs/instant-ngp/issues/119" target="_blank" rel="noopener nofollow ugc">error: parameter packs not expanded with ‘...’</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-02-07" data-time="13:34:00" data-timezone="UTC">01:34PM - 07 Feb 22 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2022-02-10" data-time="09:41:09" data-timezone="UTC">09:41AM - 10 Feb 22 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/duckworthd" target="_blank" rel="noopener nofollow ugc">
          <img alt="duckworthd" src="https://avatars.githubusercontent.com/u/626236?v=4" class="onebox-avatar-inline" width="20" height="20">
          duckworthd
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">I suspect this issue is not directly related to instant-ngp's code, but I'll pos<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">t it here all the same in case anyone else sees a similar issue.

TL;DR there's an issue compiling `std_function.h`, a dependency of `tiny-cuda-nn`.

```
&gt; cmake --build build --config RelWithDebInfo

Consolidate compiler generated dependencies of target tiny-cuda-nn
[  1%] Building CUDA object dependencies/tiny-cuda-nn/src/CMakeFiles/tiny-cuda-nn.dir/common.cu.o
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
  435 |         function(_Functor&amp;&amp; __f)
      |                                                                                                                                                 ^
/usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
  530 |         operator=(_Functor&amp;&amp; __f)
      |                                                                                                                                                  ^
/usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
gmake[2]: *** [dependencies/tiny-cuda-nn/src/CMakeFiles/tiny-cuda-nn.dir/build.make:76: dependencies/tiny-cuda-nn/src/CMakeFiles/tiny-cuda-nn.dir/common.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:306: dependencies/tiny-cuda-nn/src/CMakeFiles/tiny-cuda-nn.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2
```

Here's a bit more info on my setup. I'm using CUDA v11.4.4 and GCC v.11.2.0 on Debian. I saw an identical issue when compiling with CUDA v11.6.0.

```
&gt; nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_Oct_11_21:27:02_PDT_2021
Cuda compilation tools, release 11.4, V11.4.152
Build cuda_11.4.r11.4/compiler.30521435_0

&gt; gcc --version
gcc (Debian 11.2.0-13) 11.2.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
```

Any tips appreciated!</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>It works with older version of GCC but since 11.3 is the default shipped with ubuntu 22 it will be a pain for users to have to downgrade their compiler to install a python package.</p>
<p>Do you think it would be simpler to ditch the interface with pytorch and maybe use cupy or pycuda ?</p>

ptrblck: <aside class="quote no-group" data-username="GuillaumeLeclerc" data-post="4" data-topic="173172">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/guillaumeleclerc/48/1445_2.png" class="avatar"> GuillaumeLeclerc:</div>
<blockquote>
<p>So statically link against CUDA and dynamic link against pytorch ?</p>
</blockquote>
</aside>
<p>Statically linking the CUDA <em>runtime</em> is the right approach. I don’t know which additional CUDA libraries you would need.</p>
<aside class="quote no-group" data-username="GuillaumeLeclerc" data-post="4" data-topic="173172">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/guillaumeleclerc/48/1445_2.png" class="avatar"> GuillaumeLeclerc:</div>
<blockquote>
<p>What if pytorch upgrades ?</p>
</blockquote>
</aside>
<p>In that case your package might need to be updated as well assuming it depends on PyTorch directly. If you don’t use any PyTorch operations, your updates would be more flexible.<br>
You can also take a look at e.g. <a href="https://github.com/pyg-team/pytorch_geometric"><code>pytorch_geometric</code></a> which should have a similar release process you are targeting.</p>
<aside class="quote no-group quote-modified" data-username="GuillaumeLeclerc" data-post="5" data-topic="173172">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/guillaumeleclerc/48/1445_2.png" class="avatar"> GuillaumeLeclerc:</div>
<blockquote>
<p>I can’t even have it compile reliably at install time with GCC 11.3 and the latest version of pytorch it fails with this issue:<br>
It works with older version of GCC but since 11.3 is the default shipped with ubuntu 22 it will be a pain for users to have to downgrade their compiler to install a python package.</p>
</blockquote>
</aside>
<p>I don’t fully understand why users would need to build your package if you are planning to release wheels.</p>
<aside class="quote no-group" data-username="GuillaumeLeclerc" data-post="5" data-topic="173172">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/guillaumeleclerc/48/1445_2.png" class="avatar"> GuillaumeLeclerc:</div>
<blockquote>
<p>Do you think it would be simpler to ditch the interface with pytorch and maybe use cupy or pycuda ?</p>
</blockquote>
</aside>
<p>I think it depends on your use case and what your package is supposed to do.<br>
If you don’t have any dependency on PyTorch and are not using any of its operations, you might not want to add it as a dependency and e.g. using <code>pycuda</code> might be easier.</p>