Question about model behavior with DistributedDataParallel

username: Philipp_Singer
cooked: <p>Hi,</p>
<p>I have a short question about the behavior of the initial model initialization with DDP.</p>
<p>Let’s say I use two GPUs, and my code looks something like this:</p>
<pre><code class="lang-auto">model = Model().to(device)
model = DistributedDataParallel(model)
</code></pre>
<p>My question now is whether the model is initialized two times separately, or whether the model as initialized on rank 0 is mirrored on rank 1.</p>
<p>So are both GPUs starting from the same initialization or from a different one assuming the seeds are different?</p>

username: ssgosh
cooked: <p>Per my understanding, the model is already initialized upon your call to <code>Model()</code>. Then <code>.to(device)</code> will move it to the specified device (which will probably be the first gpu on the first machine).</p>
<p>The call <code>DistributedDataParallel(model)</code> will replicate your model’s params on the machines and devices you’ve specified.</p>
<p>So it starts from the same initiaization. Moreover, the computed gradients for your model’s params on each machine/device will be sent back to the first machine/device and summed.</p>

username: Philipp_Singer
cooked: <p>But <code>.to(device)</code> is called two times where device is always the gpu+rank. So the model is sent with two different initializations at first to both gpus, but what I am wondering now is if <code>DistributedDataParallel(model)</code> syncs the initialization from rank 0 to rank 1?</p>

username: ssgosh
cooked: <p>Not sure where <code>.to(device)</code> is getting called twice. I believe we are supposed to call it only once, for just one device.</p>
<p>I have used <code>DataParallel</code> and not <code>DistributedDataParallel</code>, but I believe the semantics are the same. In my usage, I call <code>model.to(device)</code> only once (where device is <code>cuda:0</code>). After that, I do <code>dpmodel = DataParallel(model)</code> only once. This replicates the model onto both devices.</p>
<p>The <code>input</code> is also on the original device (<code>cuda:0</code>). Now, if I perform <code>output = dpmodel(input)</code>, then the half of the input batch gets replicated to <code>cuda:1</code>. The computation is performed in parallel on both GPUs, and then <code>output</code> is computed by concatenating the two half-batches. <code>output</code> is on the same device as <code>input</code>.</p>

username: ssgosh
cooked: <p>Perhaps my replies are not applicable. DistributedDataParallel seems much more complicated :-). Please ignore.</p>

username: Philipp_Singer
cooked: <p>Yeah, DP behaves very differently than DDP. So your reply does not really help me, but thanks for the efforts.</p>
<p>Hope someone who knows the details about DDP can help me</p>

username: ssgosh
cooked: <p>From <a href="https://pytorch.org/docs/stable/notes/ddp.html#internal-design" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch 2.1 documentation</a> , under Internal Design</p>
<blockquote>
<p><strong>Construction</strong>: The DDP constructor takes a reference to the local module, and broadcasts <code>state_dict()</code> from the process with rank 0 to all other processes in the group to make sure that all model replicas start from the exact same state.</p>
</blockquote>
<p>Hope this helps.</p>