First torch import and .to(device) very slow when using a cached conda package

MiguelJaques: <p>I’m running into this rather odd behaviour.</p>
<p>As per the title: the first torch import and .to(device) is very slow (upwards of 2 minutes) when using an environment where pytorch was installed from the conda cache (i.e., it wasn’t downloaded).</p>
<p>This does not happen if conda has to download pytorch (e.g. when I’m using a new version that hadn’t been installed yet).</p>
<p>This seems to happen regardless of pytorch version or GPU type. I’m using AWS images (both my own and the AWS Deep Learning Base AMI). This happens for both.</p>
<p>In fact, AWS Deep Learning Base AMI already comes with the environments ready to activate, and the first import still takes a long time.</p>
<p>Any idea as to why freshly downloaded pytorch runs quickly at the first call, whereas cached/preinstalled pytorch doesn’t?</p>

ptrblck: <p>Could you post the installed PyTorch version, how you’ve installed it, which CUDA toolkit (or runtime in the binary) and OS you were using, and which GPU?<br>
The issue description would point towards a call into the CUDA JIT compiler, which would be triggered to compile the kernels for a missing GPU architecture.</p>

MiguelJaques: <p>This has happened with conda installed pytorch=&lt;1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1&gt;, cudatoolkit=&lt;10.1,10.2,11.0,11.1&gt;, with ubuntu18. nvidia-smi shows the driver is version 460.</p>
<p>This happens with K80, T4, and V100 GPUs.</p>
<p>The puzzling bit is the fact that if conda has to download the package, it imports fast; if it uses the cached package, it takes a long time.</p>

ptrblck: <p>Thanks for the information. This sounds rather like a installation issue with the conda cache and I don’t know what might be causing it.</p>

sonnguyen: <p>I had the same problem.<br>
I think the reason because conda does not see <em>torch</em> location (when I do “conda list” there is no <em>torch</em> package, eventhough <em>torchvison</em> and <em>torchaudio</em> are there).<br>
I tried to install <em>torch</em> again using pip - in stead of <em>conda install</em>, the problem disappears. I don’t know if it will cause any incompatibility, but it solved my “slow import torch” problem.</p>

ptrblck: <p>Did you by chance check the <code>torch.__path__</code> after importing it in the “slow” approach?<br>
Based on your description it seems that your conda environment didn’t even had <code>torch</code> as a package in its list, so it would be interesting where it’s even able to find it.<br>
I guess it’s (somehow) finding an older installation, which might trigger the CUDA JIT for your compute capability, but I would rather expect conda to raise an error of a missing PyTorch installation.</p>

sonnguyen: <p>Sorry, I didn’t check the torch.<strong>path</strong>, and I don’t want to roll back to the “slow” approach. It’s strange that</p>
<p><code> conda install torch</code></p>
<p>successfully, but <code>conda list</code> does not show <code>torch</code> . I guess it’s a <code>conda</code> issue.</p>

sonnguyen: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
I have 2 conda environments, one with <code>torch v1.9.0+cu111</code> still has slow import problem, the other one with <code>torch v0.4.1</code> is fine.<br>
<code>conda list</code> shows <code>torch</code> in both and<br>
<code>torch.__path__</code> are similar in both environments which is<br>
<code>CondaEnvironments/EnvName/lib/python3.7/site-packages/torch</code>.<br>
Does it give you any hint about what’s wrong?</p>

sonnguyen: <p>I think the problem is with cuda version. I have that problem with PyTorch1.9 + cuda11, it dissappeared with PyTorch1.9 + cuda10</p>

ptrblck: <p>I don’t believe it’s a pure binary issue, as I haven’t seen this behavior on any system I’m using.<br>
Based on your previous description:</p>
<aside class="quote no-group" data-username="sonnguyen" data-post="7" data-topic="117510">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sonnguyen/48/13914_2.png" class="avatar"> sonnguyen:</div>
<blockquote>
<p>It’s strange that</p>
<p><code> conda install torch</code></p>
<p>successfully, but <code>conda list</code> does not show <code>torch</code> . I guess it’s a <code>conda</code> issue.</p>
</blockquote>
</aside>
<p>I would still assume that conda finds multiple installations and triggers the CUDA JIT compiler for missing architectures.<br>
Let me know, if you have a way to reproduce this issue.</p>

sonnguyen: <p>I think you are right.<br>
If I load cuda11 before import torch, the problem dissappers. Which is weird since torch binay installed its own cuda, I didn’t have to load cuda separately ever before.<br>
It’s a little complicated to reproduce, I’m running torch in an HPC which has some dependencies in path of packages.<br>
Thank you very much for your comments, adding one more line of code to load cuda11 is trivial.</p>