A question about DDP tutorial

username: Frazer
cooked: <aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" target="_blank" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.7.0...</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
I have read the tutorial and I run the demo_basic with 2 visible gpus. In the function, I output the device of <strong>labels</strong> and <strong>outputs</strong>, and they are on gpus, while the inputs of <strong>torch.randn(20, 10)</strong> is on cpu. The tutorial is a little confused that the backend is gloo, but the main function tries to use gpus.<br>
My question is that why don’t we need to transfer the inputs to gpus ?<br>
Also, the tutorial might be more practical if dummy datasets and dataloaders are provided.
<pre><code class="lang-auto">def demo_basic(rank, world_size):
    print(f"Running basic DDP example on rank {rank}.")
    setup(rank, world_size)

    # create model and move it to GPU with id rank
    model = ToyModel().to(rank)
    ddp_model = DDP(model, device_ids=[rank])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    optimizer.zero_grad()
    outputs = ddp_model(torch.randn(20, 10))
    labels = torch.randn(20, 5).to(rank)
    loss_fn(outputs, labels).backward()
    optimizer.step()

    cleanup()
</code></pre>

username: mrshenli
cooked: <blockquote>
<p>My question is that why don’t we need to transfer the inputs to gpus ?</p>
</blockquote>
<p>Thanks for pointing this out. Yep, it’s a good practice to also move the inputs to the destination GPU. The reason that the tutorial code didn’t fail is because DDP will recursively detect tensors in the inputs and move them to the target device. See the code below:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/7df84452423f44ebe1db40a2e3463066bf954f95/torch/nn/parallel/distributed.py#L683-L684">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/7df84452423f44ebe1db40a2e3463066bf954f95/torch/nn/parallel/distributed.py#L683-L684" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/7df84452423f44ebe1db40a2e3463066bf954f95/torch/nn/parallel/distributed.py#L683-L684" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/7df84452423f44ebe1db40a2e3463066bf954f95/torch/nn/parallel/distributed.py#L683-L684</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="683" style="counter-reset: li-counter 682 ;">
          <li>inputs, kwargs = self.to_kwargs(inputs, kwargs, self.device_ids[0])</li>
          <li>output = self.module(*inputs[0], **kwargs[0])</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Frazer
cooked: <p>Thanks for the quick reply. My further question is that, what is the number of <strong>self.device_ids</strong>, when I use 2 gpus?</p>
<pre><code class="lang-auto">if len(self.device_ids) == 1:
    inputs, kwargs = self.to_kwargs(inputs, kwargs, self.device_ids[0])
    output = self.module(*inputs[0], **kwargs[0])
else:
    inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
    outputs = self.parallel_apply(self._module_copies[:len(inputs)], inputs, kwargs)
    output = self.gather(outputs, self.output_device)
</code></pre>

username: osalpekar
cooked: <p><a class="mention" href="/u/frazer">@Frazer</a> You do not need to change the <code>device_ids </code> argument passed to DDP in the distributed tutorial even if you use 2 GPUs. The <code>demo_basic</code> function is run on each spawned process, so each process ends up with a replica of ToyModel wrapped by DDP.</p>

username: Frazer
cooked: <p><a class="mention" href="/u/osalpekar">@osalpekar</a> Yeah, I know and I don’t want to change <code>device_ids</code>.  I just want to know the flow of the program. I guess you mean <code>len(self.device_ids) == 1</code> with 2 gpus. So when will the <code>else</code> condition be triggered ?</p>

username: osalpekar
cooked: <p>Ah I see, thanks for the clarification! The else condition will be triggered when you pass a list of multiple ranks as the <code>device_ids</code> arg to the DDP constructor. This basically allows you to specify which CUDA devices model replicas will be placed on (docs <a href="https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html" rel="noopener nofollow ugc">here</a>).</p>

username: mrshenli
cooked: <p>+1 to <a class="mention" href="/u/osalpekar">@osalpekar</a>’s comment.</p>
<p>One thing I want to add is that, when DDP was initially introduced, it has two modes:</p>
<ul>
<li>single-process multi-device (SPMD): each process exclusively works on multiple GPUs, and hence there will be multiple model replicas within the same process. In this case, the <code>device_ids</code> should be a list of GPUs one process should use.</li>
<li>single-process single-device (SPSD): each process exclusively works on one GPU, i.e., each process works on a single model replica. In this case, <code>device_ids</code> should only contain a single device.</li>
</ul>
<p>As SPSD is almost always the recommended way to use DDP due to perf reasons, we are planning to retire SPMD mode soon. If there are concerns, please comment here: <a href="https://github.com/pytorch/pytorch/issues/47012" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/47012</a></p>