Is torch.tensor(x, device='cuda') blocking?

username: nlgranger
cooked: <p>During a profiling session, I got the impression that <code>torch.tensor(x, device='cuda')</code> is blocking. That is, the python script only carries on after the data is effectively on GPU. If an experiment has many small kernels, it can be difficult to refill the GPU queue and get GPU utilization up again.</p>
<p>If <code>torch.tensor</code> is indeed blocking, one should use <code>torch.tensor(x, pin_memory=True).to('cuda')</code> to avoid a synchronization point.</p>
<p>I have not been able to showcase this behaviour on a toy example, and I don’t know where to look into the PyTorch code. If someone savvy could provide some clarification, that would be great.</p>
<p>Related issues:</p><aside class="onebox githubpullrequest" data-onebox-src="https://github.com/pytorch/vision/pull/7754">
  <header class="source">

      <a href="https://github.com/pytorch/vision/pull/7754" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/vision</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">



    <div class="github-icon-container" title="Pull Request">
      <svg width="60" height="60" class="github-icon" viewBox="0 0 12 16" aria-hidden="true"><path fill-rule="evenodd" d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
    </div>

  <div class="github-info-container">



      <h4>
        <a href="https://github.com/pytorch/vision/pull/7754" target="_blank" rel="noopener nofollow ugc">Replace stack/mask/reduce by indexing in _hsv2rgb</a>
      </h4>

    <div class="branches">
      <code>pytorch:main</code> ← <code>nlgranger:fix_7753</code>
    </div>

      <div class="github-info">
        <div class="date">
          opened <span class="discourse-local-date" data-format="ll" data-date="2023-07-24" data-time="09:29:55" data-timezone="UTC">09:29AM - 24 Jul 23 UTC</span>
        </div>

        <div class="user">
          <a href="https://github.com/nlgranger" target="_blank" rel="noopener nofollow ugc">
            <img alt="nlgranger" src="https://avatars.githubusercontent.com/u/3764009?v=4" class="onebox-avatar-inline" width="20" height="20">
            nlgranger
          </a>
        </div>

        <div class="lines" title="3 commits changed 1 files with 16 additions and 8 deletions">
          <a href="https://github.com/pytorch/vision/pull/7754/files" target="_blank" rel="noopener nofollow ugc">
            <span class="added">+16</span>
            <span class="removed">-8</span>
          </a>
        </div>
      </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Fixes #7753</p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubpullrequest" data-onebox-src="https://github.com/pytorch/vision/pull/7506">
  <header class="source">

      <a href="https://github.com/pytorch/vision/pull/7506" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/vision</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">



    <div class="github-icon-container" title="Pull Request">
      <svg width="60" height="60" class="github-icon" viewBox="0 0 12 16" aria-hidden="true"><path fill-rule="evenodd" d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
    </div>

  <div class="github-info-container">



      <h4>
        <a href="https://github.com/pytorch/vision/pull/7506" target="_blank" rel="noopener nofollow ugc">make device transfers non-blocking in transforms</a>
      </h4>

    <div class="branches">
      <code>pytorch:main</code> ← <code>nlgranger:fix_7504</code>
    </div>

      <div class="github-info">
        <div class="date">
          opened <span class="discourse-local-date" data-format="ll" data-date="2023-04-07" data-time="13:57:50" data-timezone="UTC">01:57PM - 07 Apr 23 UTC</span>
        </div>

        <div class="user">
          <a href="https://github.com/nlgranger" target="_blank" rel="noopener nofollow ugc">
            <img alt="nlgranger" src="https://avatars.githubusercontent.com/u/3764009?v=4" class="onebox-avatar-inline" width="20" height="20">
            nlgranger
          </a>
        </div>

        <div class="lines" title="1 commits changed 6 files with 132 additions and 54 deletions">
          <a href="https://github.com/pytorch/vision/pull/7506/files" target="_blank" rel="noopener nofollow ugc">
            <span class="added">+132</span>
            <span class="removed">-54</span>
          </a>
        </div>
      </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Fixes #7504</p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/98673">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/98673" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/98673" target="_blank" rel="noopener nofollow ugc">[ux] Non-blocking tensor constructors</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2023-04-08" data-time="12:45:28" data-timezone="UTC">12:45PM - 08 Apr 23 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/vadimkantorov" target="_blank" rel="noopener nofollow ugc">
          <img alt="vadimkantorov" src="https://avatars.githubusercontent.com/u/1041752?v=4" class="onebox-avatar-inline" width="20" height="20">
          vadimkantorov
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          enhancement
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          has workaround
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: tensor creation
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🚀 The feature, motivation and pitch

It appears that regular `torch.tensor<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">`/`torch.as_tensor` constructors incur blocking behavior when placing the result in CUDA memory: https://github.com/pytorch/vision/issues/7504 https://github.com/pytorch/vision/pull/7506

So @nlgranger had to replace `torch.tensor(..., device = my_cuda_device)` by `torch.tensor(...).to(device = my_cuda_device, non_blocking = True)`

I think, it would be more idiomatic/clean to allow `non_blocking` argument directly on torch.tensor/torch.as_tensor

### Alternatives

_No response_

### Additional context

_No response_

cc @gchanan @mruberry</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: KFrank
cooked: <p>Hi Nicolas!</p>
<aside class="quote no-group" data-username="nlgranger" data-post="1" data-topic="185563" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nlgranger/48/3607_2.png" class="avatar"> nlgranger:</div>
<blockquote>
<p>During a profiling session, I got the impression that <code>torch.tensor(x, device='cuda')</code> is blocking. That is, the python script only carries on after the data is effectively on GPU.</p>
</blockquote>
</aside>
<p>This is to be expected (at least by me).  If <code>x</code> resides on the cpu, then the<br>
cpu will be busy and “block” until it is done with its part of copying the data<br>
to the gpu.</p>
<p>But I believe that <code>torch.tensor()</code> is non-blocking in the sense that it will<br>
return before the gpu is done with whatever work it has to do.  However,<br>
this non-blocking behavior is hidden because the cpu data-transfer time is<br>
the bottleneck compared to any pure gpu work (at least on my test system).</p>
<p>The non-blocking behavior becomes apparent when <code>x</code> itself resides on the<br>
gpu (or presumably on a second gpu).</p>
<p>Here is a timing script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)
print (torch.version.cuda)
print (torch.cuda.get_device_name())

import time

import warnings
warnings.filterwarnings ('ignore')   # clean up output

for  source_device in ('cpu', 'cuda'):
    print ('source_device:', source_device)
    
    for  iMeg in (100, 200, 400, 800):
        if  source_device == 'cuda' and iMeg &gt; 400:   # avoid out of memory
            break
        t_source = torch.randn (iMeg, 1000, 1000, device = source_device)
        
        # warmup
        tc0 = torch.tensor (t_source, device = 'cuda')
        tc0 = None
        
        torch.cuda.synchronize()   # make sure gpu is ready
        t0 = time.time()
        tc1 = torch.tensor (t_source, device = 'cuda')
        t1 = time.time()
        print ('iMeg:', iMeg, '  t_nosync: ', t1 - t0)
        
        torch.cuda.synchronize()
        tc1 = None
        
        torch.cuda.synchronize()   # make sure gpu is ready
        t0 = time.time()
        tc2 = torch.tensor (t_source, device = 'cuda')
        torch.cuda.synchronize()   # wait for torch.tensor() to actually finish
        t1 = time.time()
        print ('iMeg:', iMeg, '  t_sync:   ', t1 - t0)
        
        tc2 = None

        t_source = None

</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">2.0.1
11.8
GeForce GTX 1050 Ti
source_device: cpu
iMeg: 100   t_nosync:  0.06376481056213379
iMeg: 100   t_sync:    0.06362557411193848
iMeg: 200   t_nosync:  0.12702536582946777
iMeg: 200   t_sync:    0.1270275115966797
iMeg: 400   t_nosync:  0.2543323040008545
iMeg: 400   t_sync:    0.25426554679870605
iMeg: 800   t_nosync:  0.5085372924804688
iMeg: 800   t_sync:    0.5087063312530518
source_device: cuda
iMeg: 100   t_nosync:  3.981590270996094e-05
iMeg: 100   t_sync:    0.008202552795410156
iMeg: 200   t_nosync:  3.6716461181640625e-05
iMeg: 200   t_sync:    0.016355037689208984
iMeg: 400   t_nosync:  3.719329833984375e-05
iMeg: 400   t_sync:    0.03267312049865723
</code></pre>
<p>The fact that the gpu → gpu “nosync” timings are much shorter than the<br>
analogous “sync” timings shows that <code>torch.tensor (x, device = 'cuda')</code><br>
returns asynchronously when <code>x</code> resides on the gpu.</p>
<p>Best.</p>
<p>K. Frank</p>

username: nlgranger
cooked: <blockquote>
<p>the cpu data-transfer time is the bottleneck compared to any pure gpu work (at least on my test system).</p>
</blockquote>
<p>That’s my problem, it did not seem to be the case on a compute node with A100 and a fairly large model, but just like you, I cannot reproduce it locally.</p>