How to multi-node parallel in dockers（container）？

HanSheng_Huang: <p>there are two machine <strong>A，B</strong>. There are two containers in the two machines, namely <strong>ContainerX, ContainerY</strong>. I set node <strong>A(containerX)</strong> as <strong>master</strong>.<br>
their IP addresses are</p>
<pre><code class="lang-auto">A : 192.168.1.1 ， X：10.10.1.1
B：192.168.2.2， Y：10.10.2.2
</code></pre>
<p>and I add port forwarding. 1234 port access in host will forward to the container port 1234 in it.<br>
like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23943678fdb3f7e1330383cc39ad4aa9e8e19cc5.png" data-download-href="https://discuss.pytorch.org/uploads/default/23943678fdb3f7e1330383cc39ad4aa9e8e19cc5" title="ip and topology"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23943678fdb3f7e1330383cc39ad4aa9e8e19cc5_2_690x271.png" alt="ip and topology" data-base62-sha1="54KfRDOezN5oWePLVCOByNokU4J" width="690" height="271" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23943678fdb3f7e1330383cc39ad4aa9e8e19cc5_2_690x271.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23943678fdb3f7e1330383cc39ad4aa9e8e19cc5_2_1035x406.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23943678fdb3f7e1330383cc39ad4aa9e8e19cc5_2_1380x542.png 2x" data-dominant-color="99A6D1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ip and topology</span><span class="informations">1714×675 38.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I want to run the most basic multi-node distributed parallel.</p>
<p>The dist init code is follow</p>
<pre data-code-wrap="python"><code class="lang-python">args.rank = int(os.environ["RANK"])
args.world_size = int(os.environ['WORLD_SIZE'])
args.local_rank = int(os.environ['LOCAL_RANK'])
args.dist_backend = 'nccl'
torch.distributed.init_process_group(backend=args.dist_backend, 
                    world_size=args.world_size, rank=args.rank)
</code></pre>
<p>The bash command is:<br>
In <strong>node A(container X)</strong></p>
<pre data-code-wrap="shell"><code class="lang-shell">NCCL_DEBUG=INFO torchrun \
--nnodes=2  \
--node_rank=0 \
--nproc_per_node=8 \
--master_port=1234 \
--master_addr=10.10.1.1 \ 
ddp.py
</code></pre>
<p>In <strong>node B(containerY)</strong></p>
<pre data-code-wrap="shell"><code class="lang-shell">NCCL_DEBUG=INFO torchrun \
--nnodes=2  \
--node_rank=1 \
--nproc_per_node=8 \
--master_port=1234 \
--master_addr=192.168.2.2 \ 
ddp.py
</code></pre>
<p>But It can’t work.<br>
<strong>I konw use <code>--net=host</code> can solve the problem. But some other container method (like lxd) not support it.</strong>  I want to know a more elegant way.</p>
<blockquote>
<p>Only open port 1234 not correct?<br>
I have try to modify the <strong>MASTER_ADDR in bash command</strong> (Change it to 192.168.1.1 still not work).</p>
</blockquote>

piao_A: <p>Similar issue. Wondering whether you’ve tackled it <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_tear.png?v=12" title=":smiling_face_with_tear:" class="emoji" alt=":smiling_face_with_tear:" loading="lazy" width="20" height="20"></p>

fduwjj: <p>Don’t we need to use same master_addr across hosts? IIUC</p>

fengjk12138: <p>what master_addr should be used？<br>
192.168.1.1？</p>

fduwjj: <p>as long as you use the same one for both nodes.</p>