Pytorch distributed ephemeral ports communication after rendezvous

selineni: <p>hi Shen Li/pytorch team,</p>
<pre><code> After the rendezvous,  is there a way to restrict the connection between some peers in the p2p alignment.   like for example if there are 3 peers, can i restrict communication between p1-p2 and p1-p3 exclusively.   as I see it the process automatically starts ephemeral ports over tcp to talk to all peers even though i programmatically restricted data communication between p2 and p3?  your response if greatly appreciated
</code></pre>

H-Huang: <p>Thanks for the question! Is this for RPC or for collective communication libraries (e.g. NCCL)? The rendezvous is the similar but the backend setup / communication will behave different depending on what you’re using.</p>

selineni: <p>i am using rpc. tensorpipe backend.  can i have this scenario with rpc tensorpipe backend? without connections between client1, client2, client3…  all connections should be only through central.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/14fe9f9ca5c9fec2289346665e2d7adbbf8f3b59.png" data-download-href="https://discuss.pytorch.org/uploads/default/14fe9f9ca5c9fec2289346665e2d7adbbf8f3b59" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14fe9f9ca5c9fec2289346665e2d7adbbf8f3b59_2_690x271.png" alt="image" data-base62-sha1="2ZJ4OHSf8c38ep3MfgfElIHy4AN" width="690" height="271" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14fe9f9ca5c9fec2289346665e2d7adbbf8f3b59_2_690x271.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14fe9f9ca5c9fec2289346665e2d7adbbf8f3b59_2_1035x406.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/14fe9f9ca5c9fec2289346665e2d7adbbf8f3b59_2_1380x542.png 2x" data-dominant-color="E0E0E0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2044×804 22 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

H-Huang: <p>Yes, that is possible. You can <code>init_rpc("central", rank=0, world_size=4)</code> and the rest <code>init_rpc("client1", rank=1,...)</code>, <code>init_rpc("client2", rank=2,...)</code>, and so forth</p>
<p>The only caveat is that I believe during <code>rpc.shutdown()</code> each rank will do a 1:1 communication to validate that there are no unresolved messages.</p>

selineni: <p>sure that is the problem.   in my scenario where all three clients are untrusted contractually and i cannot have any connection between client1, client2 or 3.  The only connection that is possible is with central and over tls 1.3+ with cipher.   I figured the security part.   when i analyze tcpdumps, the tensorpipe is opening some random ephemaral ports over tcp and without allowing that, the connections wont happen throwing gloo socket connection error.  Is there any other way where i can still execute pytorch rpc without connecting client1, client2 and client3</p>