Reducer Buckets message with 4 GPUs

Hiperdyne19012: <p>What does “Reducer buckets have been rebuilt in this iteration” mean?<br>
I got this at the start of training using 4 GPUs.</p>

osalpekar: <p>This refers to some of the internals of PyTorch DDP - in each backward pass, DDP must allreduce gradients across all the nodes (4 GPUs in this case) so they are all in sync we reap the benefit of using multiple GPUs. We gather these gradients into buckets (which are 25mb by default), and we initiate the allreduce once the bucket is full. Once during the course of training, these buckets are actually allocated according to when the tensors may receive grads in the backward pass. The log message you saw simply indicates this buck allocation/rebuilding process has taken place, in this case, at the start of training.</p>

PistonY: <p>Do we need release this when we finish training?</p>

H-Huang: <p><a class="mention" href="/u/pistony">@PistonY</a> The bucket lifecycle is handled for you internally.</p>

wmweng: <p>Only one GPU is used in my code, but “Reducer buckets have been rebuilt in this iteration”  is still printed at the start of training. Is it normal?</p>

clems: <p>Hi, any way one can silence this print? thx</p>

pnnaman: <p>Hi <a class="mention" href="/u/h-huang">@H-Huang</a> ! I am looking to re-implement the gradient aggregation logic for a personal research project. Could you point me to the file this is implemented in?</p>

H-Huang: <aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/reducer.cpp#L835">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/reducer.cpp#L835" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/reducer.cpp#L835" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/reducer.cpp#L835</a></h4>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="825" style="counter-reset: li-counter 824 ;">
          <li>        "disable unused parameter detection by passing the keyword argument ",</li>
          <li>        "`find_unused_parameters=False` to ",</li>
          <li>        "`torch.nn.parallel.DistributedDataParallel`. If unused parameters ",</li>
          <li>        "in the model do not change over iterations, You can try to use ",</li>
          <li>        "_set_static_graph() as a workaround if this module graph does not ",</li>
          <li>        "change during training loop.");</li>
          <li>    REDUCER_CHECK(!has_marked_unused_parameters_, logger_, common_error);</li>
          <li>  }</li>
          <li>}</li>
          <li>
          </li>
<li class="selected">void Reducer::mark_variable_ready(size_t variable_index) {</li>
          <li>  REDUCER_CHECK(</li>
          <li>      variable_index &lt; variable_locators_.size(),</li>
          <li>      logger_,</li>
          <li>      "Out of range variable index.");</li>
          <li>
          </li>
<li>  checkAndRaiseMarkedTwiceError(variable_index);</li>
          <li>  perIterationReadyParams_.insert(variable_index);</li>
          <li>  backward_stats_[variable_index] =</li>
          <li>      current_time_in_nanos() - backward_compute_start_time_;</li>
          <li>
      </li>
</ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


quornmd: <p>I get this before every batch. E.g.</p>
<pre><code class="lang-python">[2023-09-26 00:06:21,660][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:21,660][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
Epoch: [0][0/98]        Time 2.962 (2.962)      Data 0.632 (0.632)      Loss 3.1528 (3.1528)    Adv Loss (Mean) nan     Acc@1 0.088     Acc@5 0.541
[2023-09-26 00:06:22,223][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:22,223][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:22,271][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:22,271][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:22,433][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:22,433][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:22,539][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:22,539][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:22,648][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:22,648][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:22,756][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:22,756][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
Epoch: [0][5/98]        Time 0.108 (0.596)      Data 0.001 (0.109)      Loss 7.5593 (5.2144)    Adv Loss (Mean) nan     Acc@1 0.099     Acc@5 0.521
[2023-09-26 00:06:22,868][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:22,868][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:22,973][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:22,973][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:23,078][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:23,078][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:23,185][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:23,185][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:23,291][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-09-26 00:06:23,291][torch.nn.parallel.distributed][INFO] - Reducer buckets have been rebuilt in this iteration.
</code></pre>
<p>Can this cause performance issues?</p>