The GPU utilization remains unchanged during slower training

xgbj: <p>Hello everyone, we recently encountered an issue where our distributed storage system became very slow during a training process. As a result, the training speed decreased to one-fifth of its usual speed. From the monitoring data, it is evident that the main cause of this slowdown is the increased time taken for data loading, as shown in the following graph:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c35371fae3fab54e83b8ffdb096b592453d092f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0c35371fae3fab54e83b8ffdb096b592453d092f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c35371fae3fab54e83b8ffdb096b592453d092f_2_690x138.jpeg" alt="image" data-base62-sha1="1JZK3VH4nRgYXUIebf6YvDApOBp" width="690" height="138" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c35371fae3fab54e83b8ffdb096b592453d092f_2_690x138.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c35371fae3fab54e83b8ffdb096b592453d092f_2_1035x207.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c35371fae3fab54e83b8ffdb096b592453d092f_2_1380x276.jpeg 2x" data-dominant-color="F4F4F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×386 81.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Due to the synchronization of Sync-BN across multiple GPUs, the forward time of our training process has also slowed down, which is expected. However, we are curious as to why the GPU utilization has not changed significantly during this period.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/9788b772d2f102b22f5d1e664f4c79314d5ba8f7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9788b772d2f102b22f5d1e664f4c79314d5ba8f7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/9788b772d2f102b22f5d1e664f4c79314d5ba8f7_2_689x281.jpeg" alt="image" data-base62-sha1="lCx0K6nm9ONiajUiQQH1mvI5oZp" width="689" height="281" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/9788b772d2f102b22f5d1e664f4c79314d5ba8f7_2_689x281.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/9788b772d2f102b22f5d1e664f4c79314d5ba8f7_2_1033x421.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/7/9788b772d2f102b22f5d1e664f4c79314d5ba8f7.jpeg 2x" data-dominant-color="EDF1F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1279×521 119 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does anyone know why this is the case? (We speculate that one possible reason is that NCCL synchronization may lead to excessively high GPU utilization, but we couldn’t find any relevant information online.)</p>

ptrblck: <aside class="quote no-group" data-username="xgbj" data-post="1" data-topic="200379">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xgbj/48/67179_2.png" class="avatar"> xgbj:</div>
<blockquote>
<p>However, we are curious as to why the GPU utilization has not changed significantly during this period.</p>
</blockquote>
</aside>
<p>NCCL communications use HW resources and are thus showing a high GPU utilization even if the current process is only waiting.</p>

xgbj: <p>Thank you for your patient reply. So it seems that the higher GPU utilization caused by NCCL is a normal phenomenon. btw, If I want to learn more details about the behavior of NCCL using hardware resources that you mentioned in your reply, what links or documents can I refer to?</p>