How to use nn.parallel.DistributedDataParallel

mingyang94: <p>I want to use nn.parallel.DistributedDataParallel to train my model on single machine with 8 2080TI GPUs. I set distributed config as torch.distributed.init_process_group(backend=‘nccl’, init_method=‘tcp://localhost:1088’, rank=0, world_size=1)<br>
However, no GPU works in train. And if I use nn.DataParallel, this problem is not existence.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08ebb7ff6e28498bc6180db00b9b7b50f2979dc7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/08ebb7ff6e28498bc6180db00b9b7b50f2979dc7" title="WX20191014-213647%402x"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08ebb7ff6e28498bc6180db00b9b7b50f2979dc7_2_466x500.jpeg" alt="WX20191014-213647%402x" data-base62-sha1="1gUQnytBzE9ocNpAF5kfigxGcED" width="466" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08ebb7ff6e28498bc6180db00b9b7b50f2979dc7_2_466x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08ebb7ff6e28498bc6180db00b9b7b50f2979dc7_2_699x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08ebb7ff6e28498bc6180db00b9b7b50f2979dc7_2_932x1000.jpeg 2x" data-dominant-color="323232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WX20191014-213647%402x</span><span class="informations">1180×1264 411 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

mrshenli: <p>DistributedDataParallel (DDP) is multi process training. For you case, you would get best performance with 8 DDP processes, where the i-th process calls:</p>
<pre><code class="lang-python">torch.distributed.init_process_group(
    backend=‘nccl’, 
    init_method=‘tcp://localhost:1088’, 
    rank=i, 
    world_size=8
)
</code></pre>
<p>Or, you could set env vars and use <code>https://github.com/pytorch/pytorch/blob/master/torch/distributed/launch.py</code></p>

mingyang94: <p>Thanks for your reply. Now I can run my training program and GPU works, but nothing print in terminal. Why?</p>

mrshenli: <p>Can you share a code snippet? What do you expect to see in terminal?</p>

solarflarefx: <p>Hello,</p>
<p>I am in a very similar situation where I have a single node and 8 GPUs.  I used the following resource as a guideline for distributed parallel training: <a href="https://github.com/dnddnjs/pytorch-multigpu" rel="nofollow noopener">https://github.com/dnddnjs/pytorch-multigpu</a></p>
<p>I was able to run this example fine, but when I try to load the model, I get the following error:<br>
Expected tensor for argument <span class="hashtag">#1</span> ‘input’ to have the same device as tensor for argument <span class="hashtag">#2</span> ‘weight’; but device 4 does not equal 0 (while checking arguments for cudnn_convolution)</p>
<p>Could this be a problem in how I am loading the training data?</p>

Lausanne: <p>Hi, it means that your input and model weight are not on the same device, just like your input on GPU-0 while your model weight on GPU-1. Note that both input and weight must be obtained by same device. I think it may be caused by the wrong load way. Can you show your loading code or give an example?</p>

solarflarefx: <p>Thanks for the reply.  I believe I found the error.  The original code was written for a single GPU and unfortunately there were multiple places where cuda:0 was hardcoded.</p>
<p>However, I do have a followup question.  From all the examples I have seen so far, the dataloader is associated with DistributedSampler: <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/distributed.html" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/utils/data/distributed.html</a></p>
<p>In many cases the dataloader loads files in a directory.  In the case of using DDP, the DistributedSampler would be used to allocate files for each GPU such that each GPU would get a unique distribution of samples from the total dataset (I am assuming that the total data items is integer divisible by the number of GPUs).</p>
<p>In my case I am loading 3D data and then take patches of this data, which serves as the training input of the network.  So one data file corresponds to more than one training input.</p>
<p>What I do currently is that I load the data by loading the files in a folder.  Then I use a custom sampler that loads the patch indices and has an iterator that passes patches when called.  I feed this sampler to a Dataloader, where I pass in the whole data, the sampler, and batch size.  This works fine for one GPU.</p>
<p>I am now moving on to converting my code to DDP.  I could put the DistributedSampler after my custom sampler, but I worry about the idea of multiple GPUs accessing the same file (again the input is a patch and different patches can come from the same file).  Am I correct to say that this would be a problem?</p>
<p>Another approach could be to put the DistributedSampler before my current sampler.  But I am a bit unsure how to hook up this DistributedSampler to my existing code.</p>
<p>I suppose yet another method could be to bypass using torch.utils.data.distributed.DistributedSampler and perhaps instead have my initial Dataset have a <strong>getitem</strong> that distributes the files among the GPUs in a manner similar to the DistributedSampler, and then keep the rest of my hooks the same.  Or alternatively, in the main loop I could have the logic for handling the distribution of files and pass this into the spawned process.</p>
<p>Would one approach be better than others?  Or should I be using another approach altogether?  Does the DDP work properly if the code does not use the DistributedSampler?</p>

spabho: <p><a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<ol>
<li>Can you please define the terms: 1) node, 2) process in the context you are using them?</li>
<li>If I want to train my model on 4 GPUs, do you call it 4 processes? or 1 process?</li>
<li>Does <code>init_method</code> correspond to the address of my PC or to the GPU I’m accessing on a cluster?</li>
<li>In <a href="https://pytorc.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">this</a> tutorial, what were you referring to as machine?</li>
</ol>
<p><a class="mention" href="/u/mingyang94">@mingyang94</a></p>
<ol>
<li>Can you please explain how you arrived at:</li>
</ol>
<aside class="quote no-group" data-username="mingyang94" data-post="1" data-topic="58184">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/a4c791/48.png" class="avatar"> mingyang94:</div>
<blockquote>
<p>init_method</p>
</blockquote>
</aside>

mrshenli: <p>Hey <a class="mention" href="/u/spabho">@spabho</a></p>
<blockquote>
<ol>
<li>Can you please define the terms: 1) node, 2) process in the context you are using them?</li>
</ol>
</blockquote>
<p>We usually use one node/machine/server to represent one physical computer which can be equipped with multiple GPUs.</p>
<p>One process is in the context of process/thread.</p>
<blockquote>
<p>If I want to train my model on 4 GPUs, do you call it 4 processes? or 1 process?</p>
</blockquote>
<p>In this case, using 4 processes with DDP should give you the best performance.</p>
<blockquote>
<p>Does  <code>init_method</code>  correspond to the address of my PC or to the GPU I’m accessing on a cluster?</p>
</blockquote>
<p>It corresponds to the address of your PC. It is giving some information for the 4 DDP processes to perform rendezvous.</p>
<blockquote>
<p>In <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">this</a> tutorial, what were you referring to as machine?</p>
</blockquote>
<p>Machines should always refer to node/server/machine/computer.</p>
<p>To be clear, there are three concepts involved in DDP training:</p>
<ul>
<li><strong>Node/Machine/Server</strong>: a physical computer that can contain multiple GPUs. It can also talk to other node through network.</li>
<li><strong>GPU</strong>: just one GPU</li>
<li><strong>Process</strong>: Each process should run its own DDP instance. Usually Each DDP instance should exclusively operate on one GPU (if your model can fit in one GPU), and DDP instances will talk to each other through network.</li>
</ul>
<p><a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="noopener nofollow ugc">This</a> example might serve better as a starting point for DDP.</p>

rzhang63: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>I was looking at the <a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="noopener nofollow ugc">tutorial</a> you mentioned.</p>
<p>In the example, it says that</p>
<blockquote>
<p>This example uses a <a href="https://pytorch.org/docs/master/generated/torch.nn.Linear.html#torch.nn.Linear" rel="noopener nofollow ugc"> <code>torch.nn.Linear</code> </a> as the local model, wraps it with DDP, and then runs one forward pass, one backward pass, and an optimizer step on the DDP model. After that, parameters on the local model will be updated, and all models on different processes should be exactly the same.</p>
</blockquote>
<p>I’m just wondering what it means by local model, and what’s the difference between the local model and the models on different processes.</p>
<p>Thanks!</p>

mrshenli: <aside class="quote no-group" data-username="rzhang63" data-post="10" data-topic="58184">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/da6949/48.png" class="avatar"> rzhang63:</div>
<blockquote>
<p>I’m just wondering what it means by local model, and what’s the difference between the local model and the models on different processes.</p>
</blockquote>
</aside>
<p>Hey <a class="mention" href="/u/rzhang63">@rzhang63</a>, each process runs its own DDP model, which wraps the local model. DDP does not own any parameters (but it will create buffers/buckets for communication). So from the perspective of parameters, they are the same.</p>
<p>The reason for calling them local model vs DDP model is that, local model by itself does not perform communication across processes. DDP takes care of communication to make sure that different local models on difference processes are always in sync, as if all processes are operating on the same global module.</p>

rzhang63: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>Thank you for your reply. I was running the example code in the <a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="noopener nofollow ugc">tutorial</a> but I got the following error:</p>
<blockquote>
<p>AttributeError: Can’t get attribute ‘demo_basic’ on &lt;module ‘<strong>main</strong>’ (built-in)&gt;</p>
</blockquote>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/b/abd3f5c070894d72d079df2deb066c56db145cdb.png" data-download-href="https://discuss.pytorch.org/uploads/default/abd3f5c070894d72d079df2deb066c56db145cdb" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/b/abd3f5c070894d72d079df2deb066c56db145cdb.png" alt="image" data-base62-sha1="ow3LIhc9ETcCy0El0yYpCgLNTLR" width="690" height="269" data-dominant-color="171717"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1136×443 27.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Do you know why this happened?</p>

mrshenli: <aside class="quote no-group" data-username="rzhang63" data-post="12" data-topic="58184">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/da6949/48.png" class="avatar"> rzhang63:</div>
<blockquote>
<p>Thank you for your reply. I was running the example code in the <a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="noopener nofollow ugc">tutorial</a> but I got the following error:</p>
</blockquote>
</aside>
<p>The link provided above points to the DDP example, but <code>demo_basic</code> is one function from <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.1.1+cu121 documentation</a>. Are you mixing these two?</p>

rzhang63: <p>The code I used is</p>
<pre><code class="lang-auto">import os
import tempfile
import torch
import torch.distributed as dist
import torch.nn as nn
import torch.optim as optim
import torch.multiprocessing as mp

from torch.nn.parallel import DistributedDataParallel as DDP

def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    
    # initialize the process group
    dist.init_process_group("nccl", rank=rank, world_size=world_size)

def cleanup():
    dist.destroy_process_group()

def demo_basic(rank, world_size):
    print(f"Running basic DDP example on rank {rank}.")
    setup(rank, world_size)
    
    # create model and move it to GPU with id rank
    model = nn.Linear(10, 10).to(rank)
    ddp_model = DDP(model, device_ids=[rank])
    
    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)
    
    optimizer.zero_grad()
    outputs = ddp_model(torch.randn(20, 10))
    labels = torch.randn(20, 5).to(rank)
    loss_fn(outputs, labels).backward()
    optimizer.step()
    
    cleanup()

def main():
    world_size = 2
    mp.spawn(demo_basic,
        args=(world_size,),
        nprocs=world_size,
        join=True)

if __name__=="__main__":
    main()
</code></pre>

mrshenli: <p>Hey <a class="mention" href="/u/rzhang63">@rzhang63</a> sorry that I mis-read your log picture. Are you using Windows? Currently PT Distributed does not support windows yet. We have a poll here: <a href="https://github.com/pytorch/pytorch/issues/37068" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/37068</a></p>
<p>If running on Linux, it should work with a minor fix.</p>
<p>Change</p>
<pre><code class="lang-python">labels = torch.randn(20, 5).to(rank)
</code></pre>
<p>to</p>
<pre><code class="lang-python">labels = torch.randn(20, 10).to(rank)
</code></pre>

rzhang63: <p>I just switched to Linux and changed the labels to <code>labels = torch.randn(20, 10).to(rank)</code>, but I still got the following error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c860a30b5310fe1498f10eb96dbcd5dcc7936cfe.png" data-download-href="https://discuss.pytorch.org/uploads/default/c860a30b5310fe1498f10eb96dbcd5dcc7936cfe" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c860a30b5310fe1498f10eb96dbcd5dcc7936cfe_2_690x245.png" alt="image" data-base62-sha1="sACxkP1AZVt1CGx44Zd6wAvyC06" width="690" height="245" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c860a30b5310fe1498f10eb96dbcd5dcc7936cfe_2_690x245.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c860a30b5310fe1498f10eb96dbcd5dcc7936cfe_2_1035x367.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c860a30b5310fe1498f10eb96dbcd5dcc7936cfe_2_1380x490.png 2x" data-dominant-color="0F0F0F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1678×598 214 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

mrshenli: <p>This seems to be a multiprocessing pickle issue. How did you launch it. Is it sth like <code>python test.py</code> from command line or through notebook?</p>
<p>And can you confirm you see the same error even if you remove all <code>torch.distributed</code> code? Say make <code>demo_basic</code> into the following and remove other functions?</p>
<pre><code class="lang-python">def demo_basic(rank, world_size):
    pass
</code></pre>

mrshenli: <p>This looks relevant to the error you are seeing: <a href="https://github.com/ipython/ipython/issues/10894" rel="nofollow noopener">https://github.com/ipython/ipython/issues/10894</a></p>