ule. If your training program
uses GPUs for training and you would like to use [ `
torch.nn.parallel.DistributedDataParallel()  `
](generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel
"torch.nn.parallel.DistributedDataParallel") module, here is how to configure
it.

    
    
    model = torch.nn.parallel.DistributedDataParallel(model,
                                                      device_ids=[arg.local_rank],
                                                      output_device=arg.local_rank)
    

Please ensure that ` device_ids  ` argument is set to be the only GPU device
id that your code will be operating on. This is generally the local rank of
the process. In other words, the ` device_ids  ` needs to be `
[args.local_rank]  ` , and ` output_device  ` needs to be ` args.local_rank  `
in order to use this utility

5\. Another way to pass ` local_rank  ` to the subprocesses via environment
variable ` LOCAL_RANK  ` . This behavior is enabled when you launch the script
with ` --use_env=True  ` . You must adjust the subprocess example above to
replace ` args.local_rank  ` with ` os.environ['LOCAL_RANK']  ` ; the launcher
will not pass ` --local_rank  ` when you specify this flag.

Warning

` local_rank  ` is NOT globally unique: it is only unique per process on a
machine. Thus, don’t use it to decide if you should, e.g., write to a
networked filesystem. See [ https://github.com/pytorch/pytorch/issues/12042
](https://github.com/pytorch/pytorch/issues/12042) for an example of how
things can go wrong if you don’t do this correctly.

##  Spawn utility  ¶

The [ Multiprocessing package - torch.multiprocessing
](multiprocessing.html#multiprocessing-doc) package also provides a ` spawn  `
function in [ ` torch.multiprocessing.spawn()  `
](multiprocessing.html#torch.multiprocessing.spawn
"torch.multiprocessing.spawn") . This helper function can be used to spawn
multiple processes. It works by passing in the function that you want to run
and spawns N processes to run it. This can be used for multiprocess
distributed training as well.

For references on how to use it, please refer to [ PyTorch example - ImageNet
implementation ](https://github.com/pytorch/examples/tree/master/imagenet)

Note that this function requires Python 3.4 or higher.

[ Next ![](_static/images/chevron-right-orange.svg) ](distributions.html
"Probability distributions - torch.distributions") [
![](_static/images/chevron-right-orange.svg) Previous ](amp.html "Automatic
Mixed Precision package - torch.cuda.amp")

* * *

© Copyright 2019, Torch Contributors.

Built with [ Sphinx ](http://sphinx-doc.org/) using a [ theme
](https://github.com/rtfd/sphinx_rtd_theme) provided by [ Read the Docs
](https://readthedocs.org) .

  * Distributed communication package - torch.distributed 
    * Backends 
      * Backends that come with PyTorch 
      * Which backend to use? 
      * Common environment variables 
        * Choosing the network interface to use 
        * Other NCCL environment variables 
    * Basics 
    * Initialization 
      * TCP initialization 
      * Shared file-system initialization 
      * Environment variable initialization 
    * Groups 
    * Point-to-point communication 
    * Synchronous and asynchronous collective operations 
    * Collective functions 
    * Multi-GPU collective functions 
    * Third-party backends 
    * Launch utility 
    * Spawn utility 

![](https://www.googleadservices.com/pagead/conversion/795629140/?label=txkmCPmdtosBENSssfsC&guid=ON&script=0)

##  Docs

Access comprehensive developer documentation for PyTorch

[ View Docs ](https://pytorch.org/docs/stable/index.html)

##  Tutorials

Get in-depth tutorials for beginners and advanced developers

[ View Tutorials ](https://pytorch.org/tutorials)

##  Resources

Find development resources and get your questions answered

[ View Resources ](https://pytorch.org/resources)

[ ](https://pytorch.org/)

  * [ PyTorch ](https:/