Performance optimization re: CPU-GPU synchronization

Andrei_Cristea: <p>Per the <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html#avoid-unnecessary-cpu-gpu-synchronization" rel="noopener nofollow ugc">tuning guide section on CPU-GPU synchronization</a> we should try to allow the CPU to run ahead of the GPU and avoiding <code>tensor.to(device)</code> calls is a good idea. I assume one good practice is to use <code>non_blocking=True</code> in the <code>to(device)</code> calls.</p>
<ol>
<li>Is there ever a circumstance when one should avoid <code>non_blocking=True</code>? or is it <code>False</code> by default just for legacy reasons?</li>
<li>Besides <code>non_blocking=True</code>, for an image dataset (to be clear, too large to all fit onto a GPU tensor) which uses data augmentation and a dataloader with random sampling, where is the best place to run the <code>to(device)</code> call? Is it just when iterating through the data batches in the training / inference loops, or is there some opportunity for optimization by running it elsewhere?</li>
</ol>

ptrblck: <ol>
<li>
<p>Your would use page-locked (pinned) memory on the host, which is a limited resource. Once you use page-locked memory your OS won’t be able to use it anymore and depending on your system you might be running into a situation where the OS starts to use the swap and would kill the performance of your system.</p>
</li>
<li>
<p>In the common use case you would push the data to the GPU inside the <code>DataLoader</code> loop. Depending on your use case ans system you might want to check e.g. <a href="https://docs.nvidia.com/deeplearning/dali/user-guide/docs/">DALI</a> which could yield a speedup.</p>
</li>
</ol>

Andrei_Cristea: <p>Thanks for the DALI tip, I’ll have a look.</p>
<p>Regarding the page-locked memory on host:</p>
<ol>
<li>Is the concern that the memory used accumulates over time as more and more objects are placed into page-locked memory, or is the concern that a single object can be large enough to consume all the memory? Assuming the former but just verifying.</li>
<li>If it’s the former, is there any option to tell the system to unlock the previously page-locked memory at some convenient moment?</li>
<li>Does the memory hogging persist across different processes? Meaning if I run a model training script using page-locked memory, and the script stops running, should I expect the memory to then be released, or is it possible that it outlives the process that originated it?</li>
<li>Is there any relationship between <code>to(device)</code> and specifying <code>pin_memory=True</code> in a DataLoader?</li>
</ol>

ptrblck: <ol>
<li>
<p>If you are using <code>pin_memory=True</code> in the <code>DataLoader</code>, the page-locked memory should not increase as it should be used only for the current batches. The concern would be if you are generally allocating too much (e.g. a large number of workers, a huge batch size etc.) and/or if you are explicitly pinning memory on large tensors. In the end you should definitely try to use it, but should note that it <em>could</em> decrease the performance depending on your system which is why it’s not on my default.</p>
</li>
<li>
<p>The <code>DataLoader</code> should do it by default. If you’ve pinned it via <code>tensor = tensor.pin_memory()</code> you would have to <code>del</code> it eventually or override it.</p>
</li>
<li>
<p>Yes, the Python process would hold to the page-locked memory and it should be freed once the actual tensors are deleted or the Python process is stopped.</p>
</li>
<li>
<p>Yes, using <code>pin_memory=True</code> will move the data tensors on the host into page-locked memory and the <code>to('cuda')</code> operation could then use <code>non_blocking=True</code> to allow for an async host to device transfer.</p>
</li>
</ol>