Configuring nodes - DDP/RPC for Data Parallel Training

username: theflyingrahul
cooked: <p>I’m trying to reuse the servers in my university for Data Parallel Training (they’re hadoop nodes, no GPU, but the CPUs and memory are capable). I referred to <a href="https://pytorch.org/tutorials/beginner/dist_overview.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Distributed Overview — PyTorch Tutorials 1.10.0+cu102 documentation</a> which seems to be super high level, can barely get a thing. Should I use DDP/RPC? Any ideas on how/where to get started?</p>
<p>I went through the example in <a href="https://github.com/pytorch/examples/blob/master/distributed/ddp/README.md" class="inline-onebox" rel="noopener nofollow ugc">examples/README.md at master · pytorch/examples · GitHub</a> ; still not very clear!</p>
<p>I understand that I should use gloo as the backend, but how do I configure the slave nodes?<br>
I understand that process with rank 0 is the master, but where do I provide the slave IPs? How to get the slave(s) to listen to the master node for new jobs? How do I start the slave processes?</p>
<p>All of them are CentOS 7 servers, 12 cores/64GB per node, got about 12 of these.</p>
<p>Thanks in advance!</p>

username: pritamdamania87
cooked: <p>If you want to do Data Parallel Training, you should use DDP and this tutorial should help you: <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.1.1+cu121 documentation</a>.</p>
<blockquote>
<p>I understand that I should use gloo as the backend, but how do I configure the slave nodes?<br>
I understand that process with rank 0 is the master, but where do I provide the slave IPs? How to get the slave(s) to listen to the master node for new jobs? How do I start the slave processes?</p>
</blockquote>
<p>I think you probably need to familiarize yourself with ProcessGroups and our ProcessGroup API before looking into DDP. This tutorial would give a good overview: <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" class="inline-onebox" rel="noopener nofollow ugc">Writing Distributed Applications with PyTorch — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>
<p>Essentially to run DDP, first you need spawn N processes and these processes have ranks [0, N). Now for all of these processes to talk to each other, first you need to initialize a ProcessGroup (think of this as a communication channel you are initializing across all processes) using <code>init_process_group</code>.</p>
<p>Regarding initialization you can refer to this section of our docs (TCP initialization): <a href="https://pytorch.org/docs/stable/distributed.html#initialization" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 2.1 documentation</a>. You only need to specify the IP of the master on all the ranks/processes. All processes which are not rank 0 will try to connect to that IP and discover all of the other peers.</p>
<p>So essentially you start N processes and specify a master addr on all, then all processes discover each other via that master address. After that point these processes form a ProcessGroup and you can run collective operations as you wish across the entire process group.</p>

username: theflyingrahul
cooked: <p>Thank you for your inputs, that was really helpful!<br>
Got the nodes talking to each other.</p>
<p>As I’ve mentioned, I don’t have GPUs installed on these machines. In the ToyModel example, how do I alter these two lines to move the processes to the CPUs of the respective nodes?</p>
<pre><code class="lang-auto">    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

    # create model and move it to GPU (need to move to CPU) with id rank
    model = ToyModel().to(rank)
    ddp_model = DDP(model, device_ids=[rank])
</code></pre>
<p>Keeps throwing an error that “NVIDIA driver is missing”, I do not intend to use CUDA.</p>

username: pritamdamania87
cooked: <p>You can just do this:</p>
<pre><code class="lang-auto">model = ToyModel()
ddp_model = DDP(model)
</code></pre>

username: theflyingrahul
cooked: <p>Thank you, Pritam.<br>
That worked, marking it as solution now.</p>