Use two GPUs from two different virtual machines parallelly

Santhosh: <p>Does PyTorch provide any options to train a model using two GPUs installed in two different virtual machines parallelly? If so, Could someone please help me to implement it?<br>
I found that the DataParallel function is supposed to run multi GPUs present on the same machine, but couldn’t able to find any suitable information for my requirement. TIA</p>

wayi: <blockquote>
<p>Does PyTorch provide any options to train a model using two GPUs installed in two different virtual machines parallelly?</p>
</blockquote>
<p>Relevant link: <a href="https://pytorch.org/docs/stable/distributed.html#launch-utility" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 2.1 documentation</a><br>
Not sure if you can do this with virtual machines.</p>
<blockquote>
<p>I found that the DataParallel function is supposed to run multi GPUs present on the same machine, but couldn’t able to find any suitable information for my requirement.</p>
</blockquote>
<p>You are right: DataParallel is not designed for multi-node training.</p>

Santhosh: <p>Thanks for your response. I’ve checked the distributed documentation and I don’t really understand the Multi-Node multi-process distributed training section.<br>
Why --master_addr=“192.168.1.1” and --master_port=1234 arguments have the same values on both the nodes? why the IP and port values aren’t different? will it work if I assign two respective virtual machine IP addresses and port numbers in those arguments?</p>

wayi: <p>According to “Environment variable initialization” section in <a href="https://pytorch.org/docs/stable/distributed.html" rel="noopener nofollow ugc">PyTorch Distributed tutorial</a>, rank 0 node will be used to set up all connections, and both <code>MASTER_PORT</code> and <code>MASTER_ADDR</code> must be free port and address of rank 0 node. Therefore, you should always use the same values for all the nodes in the same process group.</p>

Santhosh: <p>How do the two virtual machines connect to each other and run GPUs from another machine parallelly? Out of two virtual machine, on which machine should I run the below distribution command?</p>
<p>python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_YOU_HAVE<br>
–nnodes=2 --node_rank=0 --master_addr=“192.168.1.1”<br>
–master_port=1234 YOUR_TRAINING_SCRIPT.py (–arg1 --arg2 --arg3<br>
and all other arguments of your training script)</p>
<p>python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_YOU_HAVE<br>
–nnodes=2 --node_rank=1 --master_addr=“192.168.1.1”<br>
–master_port=1234 YOUR_TRAINING_SCRIPT.py (–arg1 --arg2 --arg3<br>
and all other arguments of your training script)</p>

wayi: <blockquote>
<p>How do the two virtual machines connect to each other and run GPUs from another machine parallelly?</p>
</blockquote>
<p>Once the two machines forms a process group, you can use collective APIs like allreduce for communication.</p>
<blockquote>
<p>Out of two virtual machine, on which machine should I run the below distribution command?</p>
</blockquote>
<p>You should run these commands on two virtual machines, respectively. There is no “out of two virtual machine”.</p>

Santhosh: <aside class="quote no-group" data-username="wayi" data-post="6" data-topic="116013">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wayi/48/32374_2.png" class="avatar"> wayi:</div>
<blockquote>
<p>Once the two machines forms a process group, you can use collective APIs like allreduce for communication.</p>
</blockquote>
</aside>
<p>I don’t have an idea of the process group and allreduce API. Could you please elaborate on this process or share relevant links to follow up?</p>
<aside class="quote no-group" data-username="wayi" data-post="6" data-topic="116013">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wayi/48/32374_2.png" class="avatar"> wayi:</div>
<blockquote>
<p>You should run these commands on two virtual machines, respectively. There is no “out of two virtual machine”.</p>
</blockquote>
</aside>
<p>Do you mean to run the first distribution command on one VM and the second command on the other VM or run both the commands on both the VMs?</p>

wayi: <blockquote>
<p>I don’t have an idea of the process group and allreduce API. Could you please elaborate on this process or share relevant links to follow up?</p>
</blockquote>
<p>Check out “Collective functions” in the <a href="https://pytorch.org/docs/stable/distributed.html" rel="noopener nofollow ugc">tutorial</a>. This is lower-level API for communication. Ideally, you probably don’t need to do any explicit communication, and you can just try <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">DDP</a>, which does not let you feel any difference compared with the training on a single process.</p>
<blockquote>
<p>Do you mean to run the first distribution command on one VM and the second command on the other VM or run both the commands on both the VMs?</p>
</blockquote>
<p>Run the first distribution command on one VM and the second command on the other VM</p>