Model.cuda() takes long time

ming: <p>Even for a small model, calling its cuda() method takes minutes to finish. Is it normal? If yes what it does behind the scene that takes so long?</p>
<p>Thanks!</p>

apaszke: <p>We’re aware that there are problems with binary packages we’ve been shipping, that only appear on certain GPU architectures. Would you mind sharing what’s your GPU? I believe this has been already fixed, so a reinstall might help you.</p>
<p>Ping <a class="mention" href="/u/smth">@smth</a></p>

ming: <p>I upgraded pytorch from 0.1.6-py35_19 to 0.1.6-py35_22. It still took 2.5 minutes to run my_small_model.cuda(). I am using Conda Python 3.5 env, GTX 1080 with cuda 8.0.</p>
<p>Thanks for your quick reply!</p>

smth: <p>i will update back when this is fixed. I hope to get a fix out today or tomorrow.</p>

ming: <p>Thank you <a class="mention" href="/u/smth">@smth</a>, Looking forward to the fix!</p>

bellatoris: <p>I am using Conda  Python 2.7 env, GTX 1080 with cuda 8.0 too. And pytorch was installed by binary packages. Even when learning, it was much slower than lua’s torch.nn. Is this problem related to my GPU?  Or is it just a matter of my code?</p>
<p>Thanks.</p>

bellatoris: <p>During learning, there is a part where data calls cuda (). Does data.cuda() have same problem?</p>
<p>Thanks</p>

smth: <p>This should be fixed now if you use the command (website is updated):</p>
<pre><code class="lang-bash">conda install pytorch torchvision cuda80 -c soumith
</code></pre>

ming: <p>Tested the new release and it works great. Thanks a lot for the super responsive fix!</p>

chenyuntc: <p>This problem raises again in my GPU server. I was using Titan x, torch-0.1.11.post5-cp27,cuda8.0, install from pip using whl.<br>
But when I uninstall it and reinstall with torch-0.1.10.post2, it worked fine.</p>

stepelu: <p><a class="mention" href="/u/chenyuntc">@chenyuntc</a> check the latest comments I posted on <a href="https://github.com/pytorch/pytorch/issues/537" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/537</a>, maybe that’s the issue</p>

jtang10: <p>Hello I just encountered this problem as well. I am running pytorch on a cluster. Here is the information about the sysmtem and code I am running:</p>
<ol>
<li>Python 2.7 and CUDA 7.5 with latest pytorch.</li>
<li>GPU I am using is 1080 Ti.</li>
<li>My model is a simple bidirectional layered GRU followed by two linear layers. The Model.cuda() can take about 10 minutes. I am not sure if this is normal.</li>
</ol>
<p>Thanks in advance for any help!!!</p>

pipehappy1: <p>I don’t want to hijack the thread, but at least the title fits.</p>
<p>The case here is that the home fold where my script lives is NFS mounted from another server. When comes to model.cuda(), pytorch takes some time to move something over the NFS link.</p>
<p>But only my script is on NFS. Both pytorch under conda and the data is on local disk.</p>
<p>So I guess when pytorch compile those cuda source file it uses the same directory where the script lives in or user’s home directory?<br>
(because I run the script in local directory, model.cuda() still takes some time, thus pytorch don’t use working directory?)</p>
<p>The question is is there a way(or simple modification) to ask pytorch to use a local path to do compiling work if all the hypothesis is correct?</p>

smth: <p><a class="mention" href="/u/jtang10">@jtang10</a> if you are using 1080Ti you have to use CUDA 8.0  (and install pytorch with CUDA 8.0 version). otherwise the startup time will be very slow because CUDA 7.5 cannot support 1080Ti by default.</p>

Cognac: <p>Does CUDA 9.0 have a similar issue? I seem to have the same problem.<br>
I’m using 1080Ti, CUDA 9.0, CUDNN 7003. I’m installing PyTorch from source initially, but when I tried to install torchvision  using conda, my PyTorch installation might be overridden by conda. When I try <code>conda list</code>, I can see both</p>
<pre><code class="lang-auto">torch                     0.2.0+59e0472             &lt;pip&gt;
pytorch                   0.2.0           py36hf0d2509_4cu75    soumith
</code></pre>
<p>Running a simple script like this gives me the following results:</p>
<pre><code class="lang-auto">import torch
from datetime import datetime

for i in range(10):
    x = torch.randn(3, 4)
    t1 = datetime.now()
    x.cuda()
    print(i, datetime.now() - t1)
</code></pre>
<pre><code class="lang-auto">0 0:06:24.108245
1 0:00:00.000110
2 0:00:00.000055
3 0:00:00.000048
4 0:00:00.000046
5 0:00:00.000046
6 0:00:00.000044
7 0:00:00.000044
8 0:00:00.000044
9 0:00:00.000044
</code></pre>
<p>Sorry that I’m new to PyTorch so I might be doing something incorrectly. Thanks in advance.</p>

richard: <p>In my experience installing torchvision after installing conda overrides the pytorch source install. However you can re-install pytorch from source and you’ll be using the latest pytorch.</p>
<p><a class="mention" href="/u/cognac">@Cognac</a> what is your output of</p>
<pre><code class="lang-auto">import torch
torch.__version__
torch.version.cuda
torch.version.cudnn
</code></pre>
<p>?</p>

Cognac: <p>Thanks for the immediate response!</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.2.0_4'
&gt;&gt;&gt; torch.version.cuda
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
AttributeError: module 'torch.version' has no attribute 'cuda'
&gt;&gt;&gt; torch.version.cudnn
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
AttributeError: module 'torch.version' has no attribute 'cudnn'
</code></pre>
<p>I’ve just edited my previous post to add the information from <code>conda list</code>. For some reason I can see both <code>pytorch</code> installed from conda and <code>torch</code> from compiling the source. Am I using the wrong version?</p>

richard: <p>Interesting. You’re definitely not on pytorch master, you’re on the version that comes with conda.</p>
<p>If you want to build from source again (to see if your problem will go away on master, but I don’t know if it will), you can try the following: (I’m assuming you have the pytorch source code somewhere):</p>
<pre><code class="lang-auto">pip uninstall torch 
pip uninstall torch # yes, this is intentional
cd pytorch
python setup.py install
</code></pre>

Cognac: <aside class="quote no-group" data-username="richard" data-post="19" data-topic="102">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richard/48/1861_2.png" class="avatar"> Richard Zou:</div>
<blockquote>
<p>If you want to build from source again (to see if your problem will go away on master, but I don’t know if it will), you can try the following: (I’m assuming you have the pytorch source code somewhere):</p>
</blockquote>
</aside>
<p>Thanks, I’ll try it now. So after <code>pip uninstall torch</code> twice, should both <code>torch</code> and <code>pytorch</code> be removed from <code>conda list</code>? Do I need to remove <code>pytorch</code> as well?</p>

richard: <p>I’m trying to get torch removed from your pip list. I think you’ll still see pytorch installed via conda – you shouldn’t remove that because I believe removing that will uninstall torchvision as well.</p>