Can we measure how much time a GPU waited during all-reduce?

username: MaCasK
cooked: <p>Hi there. As all-reduce is a sync operation, if GPUs have difference in computing or i/o capability, faster GPUs will have to wait for slower ones. Is there a way to know how much time each GPU waited during an all-reduce operation?</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/macask">@MaCasK</a>  by the time someone else answers your specific query, you can take a look at this post by <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<aside class="onebox discoursetopic" data-onebox-src="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59">
  <header class="source">
      <img src="https://global.discourse-cdn.com/standard10/uploads/pytorch1/optimized/1X/6cd7da56682d360e2c6006ff3e31eb250c5a8675_2_32x32.png" class="site-icon" width="32" height="32">

      <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59" target="_blank" rel="noopener nofollow ugc" title="11:09AM - 25 January 2021">PyTorch Developer Mailing List – 25 Jan 21</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/365;"><img src="https://global.discourse-cdn.com/standard10/uploads/pytorch1/optimized/1X/b3d62d80fee5646484c48abca6207314034abab8_2_1024x542.png" class="thumbnail" width="690" height="365"></div>

<div class="title-wrapper">
  <h3><a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59" target="_blank" rel="noopener nofollow ugc">Using Nsight Systems to profile GPU workload</a></h3>
  <div class="topic-category">
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #8C6238;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">hardware-backends</span>
        </span>
      </span>
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #9EB83B;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">NVIDIA CUDA</span>
        </span>
      </span>
  </div>
</div>

  <p>This topic describes a common workflow to profile workloads on the GPU using Nsight Systems.  As an example, let’s profile the forward, backward, and optimizer.step() methods using the resnet18 model from torchvision.  To annotate each part of the...</p>

  <p>
    <span class="label1">Reading time: 2 mins 🕑</span>
      <span class="label2">Likes: 25 ❤</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: MaCasK
cooked: <p>Thanks very much, but if I’m not misunderstanding, Nsight is only giving result externally? Is there some way to get the result within the training script so training can be adjusted according to it?<br>
Also, to my knowledge Nsight is able to monitor GPU usage and I/O usage, but not for a particular process. The post you mentioned seems not so different from using time.time() or sth. else, not to seperate a all-reduce call into waiting and transferring. So I’m afraid this can’t solve my problem.<br>
Anyway, thanks to your answer.</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/macask">@MaCasK</a> yup, i know i did not answer your question <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>
<p>Are you using multi gpu in your home system or somewhere else. I am planning to buy another GPU for my personal use hence the question</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="MaCasK" data-post="3" data-topic="206134">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/e9a140/48.png" class="avatar"> MaCasK:</div>
<blockquote>
<p>Also, to my knowledge Nsight is able to monitor GPU usage and I/O usage, but not for a particular process.</p>
</blockquote>
</aside>
<p>Nsight Systems will profile all processes in a PyTorch DDP run and will show the NCCL reductions. From the timeline view you can then reason about the IDLE/wait time of different processes.</p>

username: MaCasK
cooked: <p>yes, i’m using multi gpus</p>

username: MaCasK
cooked: <p>so I need to reason it manually? or there is some way to do it automatically <strong>during training</strong>?</p>