DistributedDataParallel gradient avergaing

username: vineeths
cooked: <p>I am experimenting with gradient compression techniques for reduced communication during distributed training. However, I found out that DDP by default averages the replica gradients with all-reduce. Is there some way to “turn this OFF”, since I will be aggregating the gradients in an encoded format?</p>

username: mrshenli
cooked: <p>On the master branch, there is a prototype DDP communication hook feature, which is built for this purpose: <a href="https://github.com/pytorch/pytorch/issues/39272" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/39272</a></p>
<p>In prior releases (&lt;= v1.6), there is no way to turn gradient averaging off without modifying C++ code and recompile.</p>
<h3>Update</h3>
<p>Synced with Sinan (author of this comm hook feature), this will be reverted due to perf regression. We are investigating.</p>

username: mrshenli
cooked: <p>Examples can be found here: <a href="https://github.com/pytorch/pytorch/blob/c76fada4a859742ac679013b7428017a782e1432/torch/nn/parallel/distributed.py#L607-L684" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/c76fada4a859742ac679013b7428017a782e1432/torch/nn/parallel/distributed.py#L607-L684</a></p>
<p>IIUC, as of today, the communication bucket is still divided by the world size even if the hook is enabled. We are working on removing that division.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/c76fada4a859742ac679013b7428017a782e1432/torch/csrc/distributed/c10d/reducer.cpp#L355-L356" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/c76fada4a859742ac679013b7428017a782e1432/torch/csrc/distributed/c10d/reducer.cpp#L355-L356" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/c76fada4a859742ac679013b7428017a782e1432/torch/csrc/distributed/c10d/reducer.cpp#L355-L356</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="355" style="counter-reset: li-counter 354 ;">
<li>auto wrapped =</li>
<li>    c10::scalar_to_tensor(double(1.) / process_group_-&gt;getSize());</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: vineeths
cooked: <p>Thanks Shen Li.</p>
<p>I had in fact already seen the DDP communication hook PR and had interacted with SInan as well. I was actually looking for something more flexible which would allow me to measure the time and bits during communication.</p>
<p>I will definitely check the comm hook once ready.</p>

username: pritamdamania87
cooked: <blockquote>
<p>I was actually looking for something more flexible which would allow me to measure the time and bits during communication.</p>
</blockquote>
<p>It should be possible to do this in the current proposal of the communication hook. Could you elaborate a bit more on the limitations in the current proposal that might prevent us from doing these measurements?</p>

username: vineeths
cooked: <p>As of now, I plan to measure the time taken for gradient accumulation, and the number of bits communicated (for each iteration). I might need to even find the bits communicated for each layer in the future to explore layer wise compression.</p>

username: pritamdamania87
cooked: <blockquote>
<p>As of now, I plan to measure the time taken for gradient accumulation</p>
</blockquote>
<p>Are you referring to the <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/functions/accumulate_grad.h#L24" rel="noopener nofollow ugc">AccumulateGrad</a> function? If so, the autograd profiler would display time taken for this function.</p>
<blockquote>
<p>and the number of bits communicated (for each iteration)</p>
</blockquote>
<p>This should be possible in the current proposal of the communication hook since you can add up the bits for all the buckets.</p>
<blockquote>
<p>I might need to even find the bits communicated for each layer in the future to explore layer wise compression.</p>
</blockquote>
<p>This is probably something that you can’t do with the existing hook since it provides the entire bucket to the user and currently there is no way to split out individual parameters from the bucket.</p>