DDP, which process is doing the all_reduce to synchronize the gradients?

BruceDai003: <p>Hi, I just started to learn how to do Distributed training in pytorch. And learnt from the basic tutorials from here: <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.9.0+cu102 documentation</a> and I also read the DDP paper. But I still have some questions here.</p>
<ol>
<li>If I’m spawning 2 process on 1 machine with 2 GPUs. AFAIK, in each process, there will be a model replica during DDP construction. And the all_reduce synchronization happened during loss.backward(). So I’m wondering which of the 2 process is actually doing this reduction work? I can see that during all_reduce, we need to somehow communicate between the 2 processes. But is there like a ‘major’ process doing more work? Like doing the average of the gradients?</li>
<li>I am using GLOO currently, I just checked in <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.9.0 documentation</a>, the GLOO on GPU has limited functionality, just 2 supported functions: broadcast and all_reduce. Are these 2 functions enough for a basic DDP example on GPU to work fine?</li>
<li>Also, how could I check if my all_reduce is done on GPU, not on CPU? And besides, could I use GLOO backend to launch 2 processes on CPU to do DDP?<br>
Much appreciated, I might interpret some materials wrong, please help me out.</li>
</ol>
<p>Edit:<br>
I know I can use nvidia-smi to check my GPU usage to see if I have both GPU working. But the thing is, I have 2 different machines, one have 3 GPU, the other have 2 GPUs. I have set up to use only 2 processes in my simple DDP script. On the first machine, when running the script, I see that GPU memory usage is around 1.5G and 800M for the 2 GPUs by checking nvidia-smi. But on the second machine, with the same script, I can only see 42Mb GPU memory usage increments on both GPU cards. So that’s why I’m wondering if they actually run on GPUs, or loaded the work to CPUs internally when GPUs is somehow not configured correctly when I build pytorch from source.</p>

gcramer23: <blockquote>
<ol>
<li>If I’m spawning 2 process on 1 machine with 2 GPUs. AFAIK, in each process, there will be a model replica during DDP construction. And the all_reduce synchronization happened during loss.backward(). So I’m wondering which of the 2 process is actually doing this reduction work? I can see that during all_reduce, we need to somehow communicate between the 2 processes. But is there like a ‘major’ process doing more work? Like doing the average of the gradients?</li>
</ol>
</blockquote>
<p>These are the Gloo allreduce algorithms <a href="https://github.com/facebookincubator/gloo/blob/c22a5cfba94edf8ea4f53a174d38aa0c629d070f/gloo/allreduce.h#L38" rel="noopener nofollow ugc">https://github.com/facebookincubator/gloo/blob/c22a5cfba94edf8ea4f53a174d38aa0c629d070f/gloo/allreduce.h#L38</a></p>
<blockquote>
<ol>
<li>I am using GLOO currently, I just checked in <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.9.0 documentation</a>, the GLOO on GPU has limited functionality, just 2 supported functions: broadcast and all_reduce. Are these 2 functions enough for a basic DDP example on GPU to work fine?</li>
</ol>
</blockquote>
<p>Yes, those two functions are enough to implement a DDP algorithm. If you are doing distributed GPU training, it is recommended to use the NCCL backend.</p>
<p>More information on the distributed communication packages <a href="https://pytorch.org/docs/stable/distributed.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 2.1 documentation</a>.</p>
<blockquote>
<p>Also, how could I check if my all_reduce is done on GPU, not on CPU?</p>
</blockquote>
<p>For clarification, do you mean that the all_reduce algorithm is run on GPU?</p>
<blockquote>
<p>And besides, could I use GLOO backend to launch 2 processes on CPU to do DDP?</p>
</blockquote>
<p>Yes, initialize a process group before creating your DDP model.</p>
<p>A tutorial on DDP <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.1.1+cu121 documentation</a>.</p>

BruceDai003: <p>Thanks for quick reply.</p>
<aside class="quote no-group" data-username="gcramer23" data-post="2" data-topic="124351">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gcramer23/48/37599_2.png" class="avatar"> gcramer23:</div>
<blockquote>
<p>Yes, initialize a process group before creating your DDP model.</p>
</blockquote>
</aside>
<p>I have indeed followed the link you metioned. And to initialize a process group, I used the code:</p>
<pre><code class="lang-auto">def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # initialize the process group
    dist.init_process_group("gloo", rank=rank, world_size=world_size)
</code></pre>
<p>But I can’t tell where is the part related to using CPU GLOO instead of GPU GLOO in this initialization. Do you mean when creating the model, just like what we do without DDP, don’t use <code>model.to(device)</code> to use make the model run on CPU?<br>
For example:</p>
<pre><code class="lang-auto">    setup(rank, world_size)

    # create model and move it to CPU? with id rank
    model = ToyModel()
    ddp_model = DDP(model)
</code></pre>
<p>It seems like the <code>device_ids</code> argument is also not needed in DDP initialization, right?</p>

gcramer23: <blockquote>
<p>But I can’t tell where is the part related to using CPU GLOO instead of GPU GLOO in this initialization. Do you mean when creating the model, just like what we do without DDP, don’t use <code>model.to(device)</code> to use make the model run on CPU?</p>
</blockquote>
<p><a href="https://pytorch.org/docs/stable/distributed.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/distributed.html</a></p>
<p>" <code>torch.distributed</code> supports three built-in backends, each with different capabilities. The table below shows which functions are available for use with CPU / CUDA tensors."</p>
<p>You simply initialize the backend.  The model is on CPU unless moved to GPU.</p>
<blockquote>
<p>It seems like the device_ids argument is also not needed in DDP initialization, right?</p>
</blockquote>
<p><a href="https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html</a></p>
<p>" * <strong>device_ids</strong> (<em>list of python:int</em> <em>or</em> <a href="https://pytorch.org/docs/master/tensor_attributes.html#torch.torch.device" rel="noopener nofollow ugc"><em>torch.device</em></a>) –CUDA devices. 1) For single-device modules, <code>device_ids</code> can contain exactly one device id, which represents the only CUDA device where the input module corresponding to this process resides. Alternatively, <code>device_ids</code> can also be <code>None</code>. 2) For multi-device modules and CPU modules, <code>device_ids</code> must be <code>None</code>.When <code>device_ids</code> is <code>None</code> for both cases, both the input data for the forward pass and the actual module must be placed on the correct device. (default: <code>None</code>)"</p>

BruceDai003: <p>I see. Another question is regarding this piece of code:</p>
<pre><code class="lang-auto">    # create model and move it to GPU with id rank
    model = ToyModel().to(rank)
    ddp_model = DDP(model, device_ids=[rank])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)
    optimizer.zero_grad()
    outputs = ddp_model(torch.randn(20, 10))
    labels = torch.randn(20, 5).to(rank)
    loss_fn(outputs, labels).backward()
    optimizer.step()
</code></pre>
<p>The model has been designated to a GPU and also wrapped by DDP. But when we feed in data as in this line</p>
<pre><code class="lang-auto">outputs = ddp_model(torch.randn(20, 10))
</code></pre>
<p>Shouldn’t we use <code>torch.randn(20, 10).to(rank)</code> instead?</p>

Yanli_Zhao: <p>ddp will move input to device properly</p>

BruceDai003: <p>So, which all_reduce algorithm does gloo use? I saw there is ring, bcube, halving_doubling, etc. In addition, does the host do the all_reduce or directly on devices?</p>

Yanli_Zhao: <p>in default, gloo is using ring algorithm <a href="https://github.com/facebookincubator/gloo/blob/master/gloo/allreduce.cc" class="inline-onebox" rel="noopener nofollow ugc">gloo/allreduce.cc at master · facebookincubator/gloo · GitHub</a>.  Gloo will do the all reduce on host, but NCCL will do the all reduce on devices</p>

BruceDai003: <p>Hmm, but it seems that GLOO does support all_reduce op on CUDA tensors, right? So my question is why bother running the all_reduce on host (You have to cudaMemcpy the CUDA tensors to CPU and do the all_reduce there, and then transfer it back to GPU devices again)? Why not directly do all_reduce on GPU? It seems on GPU is faster, am I right?</p>