Using .cuda() is very slow on pytorch=0.2.0

uddeshya_upadhyay: <p>Due to compatibility issues, I am using pytorch=0.2.0 with python=2.7</p>
<p>I installed it using  <code>conda install pytorch=0.2.0 cuda80 -c soumith</code> as it was pointed out on the forum that this will lead to reduction in lag while using <code>.cuda()</code> for the first time, however I do not see any improvements and loading still takes <strong>~3mins</strong>.</p>
<p>(P.S I’m using tesla v100)</p>

rasbt: <p>Has been some time since PyTorch 0.2, but I can’t remember it being that slow. It should take a few seconds at most, not minutes. Not sure what’s causing the problem in your case, could be that it’s been a bug in PyTorch 0.2?</p>
<p>PyTorch 0.3 and 0.4 also work with Python 2.7, regardign the compatibility issues you mentioned, the Tesla V100 should work with cuda 8 &amp; 9, so I think so if you can install it  0.3 or 0.4 somehow alongside the 0.2 version, it would help figuring out whether it’s a PyTorch 0.2-specific bug or sth else.</p>

ptrblck: <p>I remember this issue occurred if a wrong CUDA version was installed and in the first run it’s recompiling pytorch for your GPU.<br>
Unfortunately there is no CUDA9 for pytorch <code>0.2.0</code>.</p>
<p>However, could you print <code>torch.version.cuda</code> after your first cuda run?</p>
<p><a class="mention" href="/u/rasbt">@rasbt</a> ideas sound good to debug your issue!</p>

uddeshya_upadhyay: <aside class="quote no-group" data-username="rasbt" data-post="2" data-topic="18662">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rasbt/48/60_2.png" class="avatar"> rasbt:</div>
<blockquote>
<p>ion, it would help figuring out whether it’s a PyTorch</p>
</blockquote>
</aside>
<p>I tried doing <code>torch.version.cuda</code> after the first <code>.cuda()</code> it seems like <code>torch.version</code> doesn’t have <code>cuda</code> attribute, it shows</p>
<pre><code class="lang-auto">Traceback (most recent call last):
File "&lt;stdin&gt;", line 1, in &lt;module&gt; 
AttributeError: 'module' object has no attribute 'cuda'
</code></pre>

hughperkins: <p>tesla v100 needs cuda 9. cuda 9 is incompatible with pytorch 0.2.0, even if you build 0.2.0 from source.</p>
<p>The solution is to upgrade to 0.4.0 (0.3.0 <em>might</em> work, but I’m like 
60%
 20% sure you need 0.4.0).</p>
<p>You’ll notice that your ~/.nv directory is probably incrasing in size without bound right?</p>

ptrblck: <p>Yeah, that was my mistake. I think the method was introduces after <code>0.2.0</code>.</p>

uddeshya_upadhyay: <p>The same lagging issue is happening on other machine as well using GTX-1060, with python 3.5.4 and<br>
<code>pytorch 0.3.0 py35cuda8.0cudnn6.0_0</code></p>
<p>the output of <code>torch.version.cuda</code> after first <code>.cuda()</code> is <code>'8.0.61'</code></p>
<p>is there any way to reduce the lag here?</p>

uddeshya_upadhyay: <p>In case the future reader is interested I solved it using the pytorch0.3.1 compiled with cuda 9 using</p>
<p><code>pip install http://download.pytorch.org/whl/cu90/torch-0.3.0.post4-cp35-cp35m-linux_x86_64.whl</code></p>

Ilya_Kotlov: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , greetings.</p>
<p>It seems that I’m facing this issue using nvidia-docker2, though that maybe you can assist to figure it out.<br>
Particularly, may it be that the nvidia-docker2 creates such recompiling because of its design ?<br>
Can you take a look <a href="https://stackoverflow.com/questions/73340841/first-commands-of-gpu-as-cuda-and-etc-running-slow-on-the-first-run-on-throug" rel="noopener nofollow ugc">stackoverflow issue</a>.</p>
<p>It’s not a pure pytorch issue, but maybe an nvidia-docker one, but maybe from a fast look at nvidia-docker design you could make some efficient conclusion for this issue in my setup and environment.</p>
<p>Particularly, do you think I should try install CUDA8.0 drivers on my host and try to see if I can make the connection between those drivers and the docker container (not sure if it is possible to manage in such way drivers, but that’s another question for CUDA I guess)?</p>
<p>Oh, forgot my Ubuntu 20 doesn’t support CUDA8.0.<br>
Well, anyway it could be useful if you could take a look and tell me on which end in that case does seem to be the problem.</p>

ptrblck: <aside class="quote no-group" data-username="Ilya_Kotlov" data-post="9" data-topic="18662">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ilya_kotlov/48/35840_2.png" class="avatar"> Ilya_Kotlov:</div>
<blockquote>
<p>Particularly, may it be that the nvidia-docker2 creates such recompiling because of its design ?</p>
</blockquote>
</aside>
<p>No, it’s not a docker feature, but the CUDA compiler will jit-compile code (if possible) if an unsupported architecture is detected.</p>
<aside class="quote no-group" data-username="Ilya_Kotlov" data-post="9" data-topic="18662">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ilya_kotlov/48/35840_2.png" class="avatar"> Ilya_Kotlov:</div>
<blockquote>
<p>Particularly, do you think I should try install CUDA8.0 drivers on my host and try to see if I can make the connection between those drivers and the docker container (not sure if it is possible to manage in such way drivers, but that’s another question for CUDA I guess)?</p>
</blockquote>
</aside>
<p>No, I would not recommend to use CUDA 8 or PyTorch 0.2, as both are now a few years old.</p>