Avoid cpu from idle while load data

username: Aakira
cooked: <p>Hello, I have a question about how the dataloader works. Suppose I have code that looks like this:</p>
<pre><code class="lang-auto">model.to("cuda")
for data, label in dataloader:
    data.to("cuda")
    label.to("cuda")

    output = model(data)  # on GPU

    loss = criterion(output, label)  # on GPU

    print(loss)  # use CPU?
</code></pre>
<p>In my dataset code, I only use the CPU. Will the line <code>loss = criterion(output, label)</code> wait for the line <code>output = model(data)</code> to finish, or will the CPU continue to the next batch and load the data? If the answer is that the CPU will wait, what should I do to avoid the situation where the CPU is idle like that?</p>

username: Tahamustapha_Nehdi
cooked: <p>Hi Aakira <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=12" title=":wave:" class="emoji" alt=":wave:" loading="lazy" width="20" height="20"><br>
The code you provided is sequential so compiler will treat it line by line no parallelism and specially asynchronous operations , but in the case where model is in gpu all asynchronous ops will hit barrier after that loss calculation will happen</p>

username: Aakira
cooked: <p>Thank you for the response. I always see everyone coding for training like in the code snippet above. However, when tracking chart of GPU usage, every time a batch of data is loaded, the GPU becomes idle (usage line drops to very low). Is there a way to help the CPU always load data ahead (or at least on time) for the GPU?</p>

username: yanzhou1994
cooked: <p>If you don’t print the loss or perform CUDA synchronization operations, the CUDA runtime will manage the copy stream corresponding to <code>data.to("cuda")</code>, as well as all the streams associated with the kernels needed for model execution. cuda runtime will handle and run these streams according to their dependency relationships.<br>
If you want to relazie more about cuda concurrently.<br>
see this ppt:</p><aside class="onebox pdf" data-onebox-src="https://developer.download.nvidia.com/CUDA/training/StreamsAndConcurrencyWebinar.pdf">
  <header class="source">

      <a href="https://developer.download.nvidia.com/CUDA/training/StreamsAndConcurrencyWebinar.pdf" target="_blank" rel="noopener nofollow ugc">developer.download.nvidia.com</a>
  </header>

  <article class="onebox-body">
    <a href="https://developer.download.nvidia.com/CUDA/training/StreamsAndConcurrencyWebinar.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://developer.download.nvidia.com/CUDA/training/StreamsAndConcurrencyWebinar.pdf" target="_blank" rel="noopener nofollow ugc">StreamsAndConcurrencyWebinar.pdf</a></h3>

  <p class="filesize">934.01 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Aakira
cooked: <p>Thank you for your response, the slides are very helpful, however, I have not grasped how your point is related to the initial question. Would you mind clarifying it further?</p>
<p>Thanks.</p>

username: yanzhou1994
cooked: <aside class="quote no-group" data-username="Aakira" data-post="3" data-topic="207962">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aakira/48/58984_2.png" class="avatar"> Aakira:</div>
<blockquote>
<p>Is there a way to help the CPU always load data ahead (or at least on time) for the GPU?</p>
</blockquote>
</aside>
<p>kernel run and cuda memory copy are all async operation.<br>
so for your code(unroll it and delete print):</p>
<pre><code class="lang-auto">data0, label0 = dataloader.next()
data0 = data0.to("cuda")
label0 = label0.to("cuda")
output0 = model(data0)
loss0 = criterion(output0, label0) 
data1, label1 = dataloader.next() # when run this line, loss0 not need to get the result and copy from device to host(it's an aysnc operation).  data1 may return before loss0 was computed.
data1 = data1.to("cuda")
label1 = label1.to("cuda")
output1 = model(data1)
loss1 = criterion(output1, label1) 

...

</code></pre>
<p>when compute output0,  loss0 (kernel running, while result not returned), the data1 maybe always returned by dataloader.<br>
In this case, CPU was not idle when model(data) was computing.</p>
<p><strong>But for modern gpu and current model, it’s always memory-bound,<br>
which means that model(data) is waiting for dataloader get data and copy to cuda device memory.</strong></p>

username: Aakira
cooked: <p>Thank you for your detailed answer, it’s very helpful.</p>