ule which call into C++ libraries. Its ` _sync_param  ` function performs intra-process parameter synchronization when one DDP process works on multiple devices, and it also broadcasts model buffers from the process with rank 0 to all other processes. The inter-process parameter synchronization happens in ` Reducer.cpp  ` . 

  * [ comm.h ](https://github.com/pytorch/pytorch/blob/v1.4.0/torch/csrc/distributed/c10d/comm.h) : implements the coalesced broadcast helper function which is invoked to broadcast model states during initialization and synchronize model buffers before the forward pass. 

  * [ reducer.h ](https://github.com/pytorch/pytorch/blob/v1.4.0/torch/csrc/distributed/c10d/comm.h) : provides the core implementation for gradient synchronization in the backward pass. It has three entry point functions: 

    * ` Reducer  ` : The constructor is called in ` distributed.py  ` which registers ` Reducer::autograd_hook()  ` to gradient accumulators. 

    * ` autograd_hook()  ` function will be invoked by the autograd engine when a gradient becomes ready. 

    * ` prepare_for_backward()  ` is called at the end of DDP forward pass in ` distributed.py  ` . It traverses the autograd graph to find unused parameters when ` find_unused_parameters  ` is set to ` True  ` in DDP constructor. 

[ ![ddp_code.png](https://user-
images.githubusercontent.com/16999635/72313120-4e7c1c80-3658-11ea-9c6d-44336b2daeac.png)
](https://user-
images.githubusercontent.com/16999635/72313120-4e7c1c80-3658-11ea-9c6d-44336b2daeac.png)

[ Next ![](../_static/images/chevron-right-orange.svg) ](extending.html
"Extending PyTorch") [ ![](../_static/images/chevron-right-orange.svg)
Previous ](cuda.html "CUDA semantics")

* * *

© Copyright 2019, Torch Contributors.

Built with [ Sphinx ](http://sphinx-doc.org/) using a [ theme
](https://github.com/rtfd/sphinx_rtd_theme) provided by [ Read the Docs
](https://readthedocs.org) .

  * Distributed Data Parallel 
    * Example 
    * Internal Design 
    * Implementation 
      * ProcessGroup 
      * DistributedDataParallel 

![](https://www.googleadservices.com/pagead/conversion/795629140/?label=txkmCPmdtosBENSssfsC&guid=ON&script=0)

##  Docs

Access comprehensive developer documentation for PyTorch

[ View Docs ](https://pytorch.org/docs/stable/index.html)

##  Tutorials

Get in-depth tutorials for beginners and advanced developers

[ View Tutorials ](https://pytorch.org/tutorials)

##  Resources

Find development resources and get your questions answered

[ View Resources ](https://pytorch.org/resources)

[ ](https://pytorch.org/)

  * [ PyTorch ](https://pytorch.org/)
  * [ Get Started ](https://pytorch.org/get-started)
  * [ Features ](https://pytorch.org/features)
  * [ Ecosystem ](https://pytorch.org/ecosystem)
  * [ Blog ](https://pytorch.org/blog/)
  * [ Contributing ](https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md)

  * [ Resources ](https://pytorch.org/resources)
  * [ Tutorials ](https://pytorch.org/tutorials)
  * [ Docs ](https://pytorch.org/docs/stable/index.html)
  * [ Discuss ](https://discuss.pytorch.org)
  * [ Github Issues ](https://github.com/pytorch/pytorch/issues)
  * [ Brand Guidelines ](https://pytorch.org/assets/brand-guidelines/PyTorch-Brand-Guidelines.pdf)

  * Stay Connected 
  * Email Address 

[ ](https://www.facebook.com/pytorch) [ ](https://twitter.com/pytorch) [
](https://www.youtube.com/pytorch)

To analyze traffic and optimize your experience, we serve cookies on this
site. By clicking or navigating, you agree to allow our usage of cookies. As
the current maintainers of this site, Facebook’s Cookies Policy applies. Learn
more, including about available controls: [ Cookies Policy
](https://www.facebook.com/policies/cookies/) .

![](../_static/images/pytorch-x.svg)

[ ](https://pytorch.org/)

  * [ Get Started ](https://pytorch.org/get-started)
  * [ Features ](https://pytorch.org/features)
  * [ Ecosystem ](https://pytorch.org/ecosystem)
  * [ Mobile ](https://