What's the benefit of relying on pytorch.hub.load() vs. publishing as a pip package?

username: pnacht
cooked: <p>I’ve started messing around with PyTorch, specifically importing models via <code>torch.hub.load()</code>. After taking a look at what the function actually does, though, I realized it basically acts as if the repo was installed as a module (it even states much in the <a href="https://pytorch.org/docs/master/hub.html#known-limitations" rel="noopener nofollow ugc">documentation</a>: “Torch hub works by importing the package as if it was installed”).</p>
<p>But… I now don’t entirely understand why PyTorch adopted this faux-install model instead of suggesting creators follow the standard Python route of publishing their models as packages on PyPI and/or Conda, especially given how the package sizes aren’t that large (the vast majority of models I’ve found download their weights from an external source instead of including them in the repo itself).</p>
<p>Basically, why did PyTorch go for:</p>
<pre data-code-wrap="python"><code class="lang-python">model = torch.hub.load("user/repo", "buzz")
</code></pre>
<p>instead of the more common:</p>
<pre data-code-wrap="python"><code class="lang-python">from repo import buzz
model = buzz()
</code></pre>
<p>Was it to keep things simple for the model creators, not having to figure out the complexities of setup.py/pyproject.toml, how to publish to the indexes, etc?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="pnacht" data-post="1" data-topic="197344">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pnacht/48/67781_2.png" class="avatar"> pnacht:</div>
<blockquote>
<p>download their weights from an external source instead of including them in the repo itself</p>
</blockquote>
</aside>
<p>Which is the correct approach as source code repositories should not store binary data apart from a few exceptions.</p>
<p><code>torch.hub</code> is an easy approach to load supported (and reviewed) models that can be easily used in e.g. CI runs without installing 3rd party dependencies. You can of course still install 3rd party libs if you prefer this approach or is a model is not available in the <code>hub</code>.</p>

username: pnacht
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="197344" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>torch.hub</code> is an easy approach to load supported (and reviewed) models</p>
</blockquote>
</aside>
<p>Ah, wait, does <code>torch.hub</code> only work with projects in the PyTorch Hub? Or, rather, the “metadata” functions such as <code>hub.list()</code> will only present projects registered in <code>pytorch/hub</code>, but AFAICT from looking at it’s <a href="https://github.com/pytorch/pytorch/blob/main/torch/hub.py#L484" rel="noopener nofollow ugc">source code</a>, <code>pytorch.load()</code> actually lets you load any repo with a <code>hubconf.py</code>.</p>
<p>And, for what it’s worth, the PyTorch Hub <a href="https://pytorch.org/docs/stable/hub.html" rel="noopener nofollow ugc">documentation</a> doesn’t suggest registering in <code>pytorch/hub</code>, just adding the <code>hubconf.py</code>.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="197344" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>can be easily used in e.g. CI runs without installing 3rd party dependencies.</p>
</blockquote>
</aside>
<p>That’s sort of my question, though… isn’t <code>torch.hub.load()</code> effectively installing a 3rd party dependency as well? I don’t see the difference between it and using <code>pip install</code>: Both download the source code, run a “setup” Python script (<code>hubconf.py</code> vs. <code>setup.py</code>) and then load the module (<code>torch.hub._load_module()</code> – which is called by <code>.load()</code> – vs. <code>from repo import model</code>).</p>

username: ptrblck
cooked: <p>You might be right and it seems I have overestimated the process of adding models into the <code>hub</code> based on your explanation. In this case, it seems it might just be a convenient wrapper to load models in your script instead of installing it from your terminal.</p>