Torch cuda installation issues and best practices when torch is a dependency

acs: <p>Hello all,</p>
<p>I have a image processing package on PyPI and some of the algorithms were implemented with TF/Keras, I decided to migrate these to torch models. I migrated everything and tested torch implementations, all work great.</p>
<p>I specify my dependencies in setup.cfg and currently torch from PyPI comes with CUDA 10.2 support. Some of my users with newer cards and drivers (like driver version 470+) face problems like <code>RuntimeError: CUDA error: no kernel image is available for execution on the device</code>. For these users removing the torch installation and re-installing it with <code>pip install torch --extra-index-url https://download.pytorch.org/whl/cu113</code> solve this problem.</p>
<p>I also have a frozen application that is made with PyInstaller where I expect to have similar problems.<br>
I know I can always ask users to install torch manually but that’s cumbersome. I want to ask if there are any better/best practices that I might be not aware of? What are the suggested details regarding having torch as a dependency and publish your packages on PyPI with cross-platform compatibility out-of-box as much as possible.</p>
<p>Any feedback/comment is welcome. Thank you.</p>

ptrblck: <p>I’m not aware of a proper way to specify these dependencies and think for now asking the user to install the appropriate binary is the way to go.<br>
However, we are right now about to remove the CUDA 10.2 binaries and host the CUDA 11.7.1 binaries on PyPI (using the <code>cuda</code> pip wheels as a dependency) so that these would be installed from the default mirror. We trying to fix the last compatibility issues and are hoping to get something ready by end of this week so that it can make it into the next stable <code>1.13.0</code> release.</p>

acs: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , thank you for prompt reply and great news regarding the new binaries. I cross-asked a related question over here: <a href="https://discuss.python.org/t/specifying-extra-index-url-in-setup-cfg-option-dependencies/19377" class="inline-onebox" rel="noopener nofollow ugc">Specifying --extra-index-url in setup.cfg option dependencies - Packaging - Discussions on Python.org</a> , if any solution comes from setuptools side, I will share it here too for archiving purposes. Meanwhile I will request from my users to install torch manually.</p>

ptrblck: <p>Let me know, if you get a response there as I would also be interested in these options. I hope we won’t need them soon, but it’s still interesting to learn about these approaches.</p>

acs: <p>I got an answer: <a href="https://discuss.python.org/t/specifying-extra-index-url-in-setup-cfg-option-dependencies/19377/2?u=flyindorito" class="inline-onebox" rel="noopener nofollow ugc">Specifying --extra-index-url in setup.cfg option dependencies - #2 by uranusjr - Packaging - Discussions on Python.org</a> .</p>
<p>I am thinking if uploading these different binaries with different version names to PyPI for pytorch can be a solution? I know you said soon CUDA 11.7.1 binaries will be hosted on PyPI but I am not confident on forward compatibility of the NVIDIA drivers. What do you think about this idea <a class="mention" href="/u/ptrblck">@ptrblck</a> ? If you are positive what I should do next for this discussion? creating an issue maybe?</p>

ptrblck: <aside class="quote no-group" data-username="acs" data-post="5" data-topic="161992">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/47e85d/48.png" class="avatar"> acs:</div>
<blockquote>
<p>I am thinking if uploading these different binaries with different version names to PyPI for pytorch can be a solution?</p>
</blockquote>
</aside>
<p>No, since the reason we cannot host them on PyPI is due to their size and it’s not a naming issue.</p>
<aside class="quote no-group" data-username="acs" data-post="5" data-topic="161992">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/47e85d/48.png" class="avatar"> acs:</div>
<blockquote>
<p>but I am not confident on forward compatibility of the NVIDIA drivers.</p>
</blockquote>
</aside>
<p>What are your concerns? Did you see any issues with it?</p>

acs: <aside class="quote no-group quote-modified" data-username="ptrblck" data-post="6" data-topic="161992" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<aside class="quote no-group" data-username="acs" data-post="5" data-topic="161992">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/47e85d/48.png" class="avatar"> acs:</div>
<blockquote>
<p>I am thinking if uploading these different binaries with different version names to PyPI for pytorch can be a solution?</p>
</blockquote>
</aside>
<p>No, since the reason we cannot host them on PyPI is due to their size and it’s not a naming issue.</p>
</blockquote>
</aside>
<p>I see, does that mean torch with CUDA 10.2 significantly smaller than torch with CUDA 11.3? Or is there a storage cap per version on PyPI?</p>
<aside class="quote no-group quote-modified" data-username="ptrblck" data-post="6" data-topic="161992" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<aside class="quote no-group" data-username="acs" data-post="5" data-topic="161992">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/47e85d/48.png" class="avatar"> acs:</div>
<blockquote>
<p>but I am not confident on forward compatibility of the NVIDIA drivers.</p>
</blockquote>
</aside>
<p>What are your concerns? Did you see any issues with it?</p>
</blockquote>
</aside>
<p>Users with pre-CUDA 11.* supporting drivers  previously reported that had runtime issues with the things I built with CUDA 11.3. Using this: <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#use-the-right-compat-package" class="inline-onebox" rel="noopener nofollow ugc">CUDA Compatibility :: NVIDIA Data Center GPU Driver Documentation</a> some of them did upgrades and were able to run the programs smoothly. I can ping you whenever another reproducible example comes by if you like.</p>

ptrblck: <aside class="quote no-group" data-username="acs" data-post="7" data-topic="161992">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/47e85d/48.png" class="avatar"> acs:</div>
<blockquote>
<p>I see, does that mean torch with CUDA 10.2 significantly smaller than torch with CUDA 11.3? Or is there a storage cap per version on PyPI?</p>
</blockquote>
</aside>
<p>Yes, CUDA 11.x is larger as it ships with more kernels for new architectures etc. While we could reduce the size a bit, it’s still way too large for us to host the PyTorch pip wheel including the CUDA runtime (in the same wheel) on PyPI. The new approach would be to build the PyTorch wheels with e.g. CUDA 11.7.1 but instead of shipping the wheel with the CUDA runtime, we would add a dependency on the CUDA pip wheels also hosted on PyPI (newly added in 11.7.1), which would reduce the PyTorch wheels significantly.</p>
<aside class="quote no-group" data-username="acs" data-post="7" data-topic="161992">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/47e85d/48.png" class="avatar"> acs:</div>
<blockquote>
<p>I can ping you whenever another reproducible example comes by if you like.</p>
</blockquote>
</aside>
<p>Yes, please do so. Minor version compatibility should work in all CUDA 11.x versions and we have to fix anything that breaks it.</p>
<aside class="quote no-group" data-username="acs" data-post="7" data-topic="161992">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/47e85d/48.png" class="avatar"> acs:</div>
<blockquote>
<p>Users with pre-CUDA 11.* supporting drivers previously reported that had runtime issues with the things I built with CUDA 11.3.</p>
</blockquote>
</aside>
<p>Note that “minor version compatibility” was added in 11.x. The linked “forward compatibility” is used for Data Center GPUs (previously called Tesla GPUs).<br>
In any case, feel free to ping me if you are seeing any issues and I can take a look.</p>