Distributed training in multinode

hariram_manohar: <p>Hi,</p>
<p>As per my knowledge with pytorch you can do parallel training on multiple GPUs/CPUs on a single node without any issue but its not matured yet to do multinode training without any issues considering asynchronous data parallelism. if its supported on multinode too please provide me a simple example to test it out.</p>
<p>Thanks.</p>

hariram_manohar: <p>Any one suggest please</p>

mrshenli: <p>Hey <a class="mention" href="/u/hariram_manohar">@hariram_manohar</a></p>
<p>There are multiple tools in PyTorch to facilitate distributed training:</p>
<ul>
<li>
<p>Distributed Data Parallel Training: checkout <a href="https://pytorch.org/docs/master/nn.html#torch.nn.parallel.DistributedDataParallel" rel="nofollow noopener">DDP</a> and <a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="nofollow noopener">this example</a> and <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="nofollow noopener">this tutorial</a></p>
</li>
<li>
<p>General Distributed Training: checkout <a href="https://pytorch.org/docs/master/rpc.html" rel="nofollow noopener">RPC</a> and <a href="https://pytorch.org/tutorials/intermediate/rpc_tutorial.html" rel="nofollow noopener">This tutorial</a></p>
</li>
</ul>
<p>BTW, for distributed training questions, please use the “distributed” tag, so that we can get back to you promptly.</p>

hariram_manohar: <p>Hi Shen Li,</p>
<p>Hope what i required is RPC as i want to do the training on multiple machines but in the tutorial or in the doc i haven’t seen anywhere mentioning about multiple servers definition where we need to execute. where do we define them ? if any detailed blog is there please let me know.</p>
<p>Basically i am trying to do training on multiple machines for the code mentioned in <a href="https://discuss.pytorch.org/t/huge-false-negatives/71239/25" class="inline-onebox">Huge False Negatives</a></p>
<p>I want to train a model for binary classification on multiple machines at same time.</p>
<p>Thanks.</p>

mrshenli: <p>Hi <a class="mention" href="/u/hariram_manohar">@hariram_manohar</a></p>
<p>Both DDP and RPC can work on multiple machines. They operate on a group of processes (called ProcessGroup), and it is up to the application how to place those processes (on a single machine or multiple machines). You just need to call <code>init_process_group</code> or <code>init_rpc</code> with proper arguments. The choice between DDP or RPC depends on whether you want to divide input data or split model.</p>
<p>Take <a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="nofollow noopener">this</a> as an example. The code launches two processes on the same machine. But you can 1) set <code>MASTER_ADDR</code> to <code>machine_1_ip</code> and <code>MASTER_PORT</code> to e.g. 23456, and 2) run the <code>example()</code> function on two machines. The two processes will rendezvous using the ip and port you provided and DDP will then handle the communication for you.</p>

hariram_manohar: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>getting below error for mentioned example, how to disable cuda.</p>
<pre><code>  File "/apps/dl/torch/ddp.py", line 20, in example
	model = nn.Linear(10, 10).to(rank)
	
	  File "/apps/miniconda3/envs/torch/lib/python3.8/site-packages/torch/cuda/__init__.py", line 94, in _check_driver
	raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
</code></pre>
<p>i haven’t installed CUDA and not required also…</p>

mrshenli: <p>To avoid using CUDA, remove all <code>.to(rank)</code> and remove <code>device_ids=[rank]</code> from DDP constructor (or set <code>device_ids=None</code>). Then DDP will only use CPU.</p>

hariram_manohar: <p>hi <a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>here are the errors i am getting.</p>
<p>Error on Master/primary server:</p>
<pre><code>Traceback (most recent call last):
  File "ddp.py", line 48, in &lt;module&gt;
	main()
  File "ddp.py", line 42, in main
	mp.spawn(example,
  File "/apps/miniconda3/envs/torch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
	while not spawn_context.join():
  File "/apps/miniconda3/envs/torch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 118, in join
	raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/apps/miniconda3/envs/torch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
	fn(i, *args)
  File "/apps/dl/torch/ddp.py", line 22, in example
	ddp_model = DDP(model)
  File "/apps/miniconda3/envs/torch/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 301, in __init__
	self._distributed_broadcast_coalesced(
  File "/apps/miniconda3/envs/torch/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 485, in _distributed_broadcast_coalesced
	dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: [/opt/conda/conda-bld/pytorch_1579022027171/work/third_party/gloo/gloo/transport/tcp/pair.cc:572] Connection closed by peer [SECONDARY_IP]:43924
</code></pre>
<p>ERROR on secondary:</p>
<pre><code>terminate called after throwing an instance of 'gloo::EnforceNotMet'
  what():  [enforce fail at /opt/conda/conda-bld/pytorch_1579022027171/work/third_party/gloo/gloo/transport/tcp/device.cc:281] rv != -1. -1 vs -1. epoll_ctl: No such file or directory
Traceback (most recent call last):
  File "ddp.py", line 48, in &lt;module&gt;
	main()
  File "ddp.py", line 42, in main
	mp.spawn(example,
  File "/apps/miniconda3/envs/torch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
	while not spawn_context.join():
  File "/apps/miniconda3/envs/torch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 105, in join
	raise Exception(
Exception: process 1 terminated with signal SIGABRT
</code></pre>
<p>here is the reference in both servers:</p>
<p>Master:</p>
<pre><code>def example(rank, world_size):
	os.environ['MASTER_ADDR'] = 'localhost'
	os.environ['MASTER_PORT'] = '12355'
</code></pre>
<p>secondary:</p>
<pre><code>def example(rank, world_size):
	os.environ['MASTER_ADDR'] = 'master_ip'
	os.environ['MASTER_PORT'] = '12355'</code></pre>

mrshenli: <ol>
<li>can you try setting <code>MASTER_ADDR</code> to the same IP?</li>
<li>Can you show the <code>ddp.py</code> is the same as the one in the <a href="https://pytorch.org/docs/master/notes/ddp.html" rel="nofollow noopener">ddp note</a>? If so, you need to modify rank, world_size, etc. accordingly.</li>
</ol>

hariram_manohar: <p>here is the ddp.py from master… other server has only difference it has master_ip address</p>
<pre><code>import torch.distributed as dist
import torch.multiprocessing as mp
import os
import torch.nn as nn
import torch.optim as optim
from torch.nn.parallel import DistributedDataParallel as DDP


def cleanup():
	dist.destroy_process_group()

def example(rank, world_size):
	os.environ['MASTER_ADDR'] = 'localhost'
	os.environ['MASTER_PORT'] = '12355'

	# create default process group
	dist.init_process_group("gloo", rank=rank, world_size=world_size)
	# create local model
	model = nn.Linear(10, 10)
	# construct DDP model
	ddp_model = DDP(model)
	# define loss function and optimizer
	loss_fn = nn.MSELoss()
	optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

	# forward pass
	outputs = ddp_model(torch.randn(20, 10))
	labels = torch.randn(20, 10)
	# backward pass
	loss_fn(outputs, labels).backward()
	# update parameters
	optimizer.step()
	print(outputs)

	cleanup()



def main():
	world_size = 2
	mp.spawn(example,
		args=(world_size,),
		nprocs=world_size,
		join=True)

if __name__=="__main__":
	main()</code></pre>

mrshenli: <p>That example code was intended to run on a single node alone. If you use it on multiple nodes, there are at least two problems:</p>
<ol>
<li>Incorrect world_size and rank: Say you run this on two different nodes. Then each would try to spawn two processes. So that you will have four process group instances, but the world_size is still two, and rank is 0, 1, 0, 1, instead of 0, 1, 2, 3, which would mess up the rendezvous algorithm. To fix this, you need to pass in correct rank and world_size for each <code>example(rank, world_size)</code> invcation.</li>
<li>Can you change the <code>MASTER_ADDR</code> to use the same IP instead of master is using <code>localhost</code> and others are using master ip?</li>
</ol>

hariram_manohar: <p>yeah i have changed from localhost to master_ip</p>
<p>is it fine if i change like this?</p>
<p>master:</p>
<pre><code>world_size = 2
mp.spawn(example,
	args=(0,1),
	nprocs=world_size,
	join=True)
</code></pre>
<p>other:</p>
<pre><code>world_size = 2
mp.spawn(example,
	args=(2,3),
	nprocs=world_size,
	join=True)</code></pre>

mrshenli: <aside class="quote no-group" data-username="hariram_manohar" data-post="12" data-topic="72110">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hariram_manohar/48/21048_2.png" class="avatar"> hariram_manohar:</div>
<blockquote>
<p>master:</p>
<pre><code class="lang-auto">world_size = 2
mp.spawn(example,
	args=(0,1),
	nprocs=world_size,
	join=True)
</code></pre>
<p>other:</p>
<pre><code class="lang-auto">world_size = 2
mp.spawn(example,
	args=(2,3),
	nprocs=world_size,
	join=True)
</code></pre>
</blockquote>
</aside>
<ol>
<li>This does not look correct to me. The <code>world_size</code> is the global size of the process group, which should be 4 in this case, but <code>nprocs</code> should be 2 on each node as each node spawns two processes.</li>
<li>The <code>args</code> doesn’t seem to match the signature <code>def example(rank, world_size)</code>. If you do this, it would pass (0, 0, 1) and (1, 0, 1) to the two processes on the first node, and (0, 2, 3) and (1, 2, 3) to the second node. But, it should be passing (0, 4), (1, 4), (2, 4), and (3, 4) to <code>example</code> Checkout the <a href="https://pytorch.org/docs/stable/multiprocessing.html#torch.multiprocessing.spawn" rel="noopener nofollow ugc"><code>multiprocessing.spawn()</code></a> API doc.</li>
</ol>
<p>To make it work on multiple nodes, you can modify the <code>example</code> signature to sth like <code>def example(local_rank, global_rank_offset, world_size)</code>, then use <code>global_rank_offset + local_rank</code> as the rank for <code>init_process_group</code>.  Then you can spawn it like:</p>
<pre data-code-wrap="python"><code class="lang-python">    world_size = 4
    global_rank_offset = 0 # this should be 2 for the other machine
    mp.spawn(example,
        args=(global_rank_offset, world_size),
        nprocs=world_size,
        join=True)
</code></pre>
<p>If you want to avoid manually configuring the rank, you can try the <code>launch.py</code> script. See <a href="https://github.com/pytorch/examples/pull/743" rel="noopener nofollow ugc">this example</a>.</p>

hariram_manohar: <p>looks like its working <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"><br>
now i want to try out for actual problem…</p>