Tensor and model .cuda() hanging indefinitely - Have to crash kernel or force close terminal

username: 3210jr
cooked: <p>Hi all,</p>
<p>Whenever I try to move my tensors or model to the GPU using either the <code>.cuda()</code> or <code>.to('cuda')</code>  method, the kernel just freezes and has to be terminated to be used again.</p>
<p>I’ve looked through several other related issues and they are either extremely old (circa 2017) or their solutions were that they had an incompatible version of cuda running - I think none were very useful.</p>
<p>Here is my environment details:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.21       Driver Version: 435.21       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 207...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   47C    P8     5W /  N/A |    303MiB /  7982MiB |      5%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1768      G   /usr/lib/xorg/Xorg                           135MiB |
|    0      2036      G   /usr/bin/gnome-shell                         116MiB |
|    0      2589      G   ...uest-channel-token=11750413998548151078    49MiB |
+-----------------------------------------------------------------------------+

</code></pre>
<p>nvcc --version</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:17_PST_2019
Cuda compilation tools, release 10.1, V10.1.105

</code></pre>
<p>And I just followed the basic installation instructions on the website:</p>
<pre><code class="lang-auto">conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

</code></pre>
<p>Any ideas? Thanks!</p>

username: tom
cooked: <p>There was an issue with cuda 10.1 minor versions between anaconda’s cuda and PyTorch’s cuda differing causing excessive JIT compiles.<br>
It is fixed and reinstalling PyTorch helps.</p>
<p>Best regards</p>
<p>Thomas</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/27807" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/27807" target="_blank" rel="nofollow noopener">Very Slow Moving Tensor to CUDA device (CUDA 10.1 with PyTorch 1.3)</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-10-12" data-time="08:55:22" data-timezone="UTC">08:55AM - 12 Oct 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-10-12" data-time="21:44:22" data-timezone="UTC">09:44PM - 12 Oct 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/dhpollack" target="_blank" rel="nofollow noopener">
          <img alt="dhpollack" src="https://avatars0.githubusercontent.com/u/368699?v=4" class="onebox-avatar-inline" width="20" height="20">
          dhpollack
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
Moving tensors to cuda devices is super slow when using pytorch 1.3 and CUDA 10.1. The issue does not occur...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">high priority</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: binaries</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: 3210jr
cooked: <p>Worked like a charm!</p>
<p>Thanks <a class="mention" href="/u/tom">@tom</a></p>