Optimizers and multiprocessing: should I manually average gradients?

bluehood: <p>Hi,<br>
similar topic to <a href="https://discuss.pytorch.org/t/can-adam-optimizer-be-used-with-torch-multiprocessing/24626">this question</a>: do optimizers work transparently in multiprocess runs or do I need to average the gradients of each process manually?</p>
<p>The <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="nofollow noopener">imagenet example</a> in the pytorch/examples repo does not do explicit gradient averaging between processes, but the <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html#distributed-training" rel="nofollow noopener">example on distributed</a> training in pytorch’s tutorials does.</p>
<p>Thanks a lot!<br>
Enrico</p>

bluehood: <p>First and last bump!</p>

jerinphilip: <p>I have a similar question <a href="https://discuss.pytorch.org/t/distributeddataparallel-loss-compute-and-backpropogation/47205">here</a>. I simultaneously opened a query in <a href="https://github.com/pytorch/fairseq/issues/779" rel="nofollow noopener">pytorch/fairseq#779</a> to which the response was that there is built in averaging.</p>
<p>How about trying some black box experiments to figure out?</p>

pietern: <p>If you use vanilla multiprocessing you’ll have to do this yourself. If you use it in combination with <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel" rel="nofollow noopener"><code>torch.nn.parallel.DistributedDataParallel</code></a> then gradient synchronization and averaging is done for you. Also see the documentation on <a href="https://pytorch.org/docs/stable/distributed.html" rel="nofollow noopener"><code>torch.distributed</code></a> for more information.</p>

makslevental: <p><a class="mention" href="/u/pietern">@pietern</a> can you show me in the source where the average is done? for the life of me i’ve been all over the codebase and i can’t find it.</p>
<p>i’m looking here</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/autograd/__init__.py#L44" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/autograd/__init__.py#L44" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/torch/autograd/__init__.py#L44</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="34" style="counter-reset: li-counter 33 ;">
<li>                    raise RuntimeError("grad can be implicitly created only for scalar outputs")</li>
<li>                new_grads.append(torch.ones_like(out))</li>
<li>            else:</li>
<li>                new_grads.append(None)</li>
<li>        else:</li>
<li>            raise TypeError("gradients can be either Tensors or None, but got " +</li>
<li>                            type(grad).__name__)</li>
<li>    return tuple(new_grads)</li>
<li>
</li>
<li>
</li>
<li class="selected">def backward(tensors, grad_tensors=None, retain_graph=None, create_graph=False, grad_variables=None):</li>
<li>    r"""Computes the sum of gradients of given tensors w.r.t. graph leaves.</li>
<li>
</li>
<li>    The graph is differentiated using the chain rule. If any of ``tensors``</li>
<li>    are non-scalar (i.e. their data has more than one element) and require</li>
<li>    gradient, then the Jacobian-vector product would be computed, in this</li>
<li>    case the function additionally requires specifying ``grad_tensors``.</li>
<li>    It should be a sequence of matching length, that contains the "vector"</li>
<li>    in the Jacobian-vector product, usually the gradient of the differentiated</li>
<li>    function w.r.t. corresponding tensors (``None`` is an acceptable value for</li>
<li>    all tensors that don't need gradient tensors).</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>and</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/python_engine.cpp#L90" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/python_engine.cpp#L90" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/torch/csrc/autograd/python_engine.cpp#L90</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="80" style="counter-reset: li-counter 79 ;">
<li>  // backwards threads hold a lock, we'll probably deadlock in the engine</li>
<li>  // destructor.</li>
<li>  if (_reinitialize_engine) {</li>
<li>    engine.~PythonEngine();</li>
<li>    new (&amp;engine) torch::autograd::python::PythonEngine();</li>
<li>    _reinitialize_engine = false;</li>
<li>  }</li>
<li>}</li>
<li>
</li>
<li>// Implementation of torch._C._EngineBase.run_backward</li>
<li class="selected">PyObject *THPEngine_run_backward(THPEngine *self, PyObject *args, PyObject *kwargs)</li>
<li>{</li>
<li>  HANDLE_TH_ERRORS</li>
<li>  _maybe_reinitialize_engine_after_fork();</li>
<li>  PyObject *tensors = nullptr;</li>
<li>  PyObject *grad_tensors = nullptr;</li>
<li>  unsigned char keep_graph = 0;</li>
<li>  unsigned char create_graph = 0;</li>
<li>  PyObject *inputs = nullptr;</li>
<li>  unsigned char allow_unreachable = 0;</li>
<li>  const char *accepted_kwargs[] = {</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


pietern: <p><a class="mention" href="/u/makslevental">@makslevental</a> It’s done not in the autograd code but in the DDP reducer code:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/8a026d4f74b71944ac2860c315996165a40f5626/torch/csrc/distributed/c10d/reducer.cpp#L321-L322" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/8a026d4f74b71944ac2860c315996165a40f5626/torch/csrc/distributed/c10d/reducer.cpp#L321-L322" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/8a026d4f74b71944ac2860c315996165a40f5626/torch/csrc/distributed/c10d/reducer.cpp#L321-L322</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="321" style="counter-reset: li-counter 320 ;">
<li>// Prescale bucket contents to turn the global sum into the global average.</li>
<li>replica.contents.div_(process_group_-&gt;getSize());</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>