How to make rpc work with WSL

frankdong: <p>I’m trying below sample on multiple machines:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/examples/blob/master/distributed/rpc/parameter_server/rpc_parameter_server.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/master/distributed/rpc/parameter_server/rpc_parameter_server.py" target="_blank" rel="nofollow noopener">pytorch/examples/blob/master/distributed/rpc/parameter_server/rpc_parameter_server.py</a></h4>
<pre><code class="lang-py">import argparse
import os
import time
from threading import Lock

import torch
import torch.distributed.autograd as dist_autograd
import torch.distributed.rpc as rpc
import torch.multiprocessing as mp
import torch.nn as nn
import torch.nn.functional as F
from torch import optim
from torch.distributed.optim import DistributedOptimizer
from torchvision import datasets, transforms

# --------- MNIST Network to train, from pytorch/examples -----


class Net(nn.Module):
    def __init__(self, num_gpus=0):
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/examples/blob/master/distributed/rpc/parameter_server/rpc_parameter_server.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>When I tried on 2 linux machine in same domain, this sample works fine. But when I try on 2 WSL machine, I can’t make these 2 WSL machine connect with each other, both machines blocking waiting on rpc init.</p>
<p>Tried the " <strong>WSL 2 TPC NETWORK FORWARDING</strong>" workaround from below post, rpc still can’t connect with each other:<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/microsoft/WSL/issues/4150" target="_blank" rel="nofollow noopener">github.com/microsoft/WSL</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/microsoft/WSL/issues/4150" target="_blank" rel="nofollow noopener">[WSL 2] NIC Bridge mode 🖧 (Has TCP Workaround🔨)</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-06-16" data-time="01:35:15" data-timezone="UTC">01:35AM - 16 Jun 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/jkasten2" target="_blank" rel="nofollow noopener">
          <img alt="jkasten2" src="https://avatars2.githubusercontent.com/u/645861?v=4" class="onebox-avatar-inline" width="20" height="20">
          jkasten2
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Issue
WSL 2 seems to NAT it's virtual network, instead of making it bridged to the host NIC. My goal is for...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">network</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>So I’m wondering if there is any working sample/setting for WSL? Any suggestions on how to trouble the issue is also highly appreciated, thanks.</p>

mrshenli: <p>We haven’t tests WSL (I assume you mean Windows Subsystem for Linux), and are not sure what are the gaps if there are any.</p>
<p>cc <a class="mention" href="/u/ptrblck">@ptrblck</a> do you know who is familiar with WSL?</p>
<p>Not sure if this can help, I would try to verify if the following command resolves to the correct interface. If not, set <a href="https://pytorch.org/docs/stable/distributed.html#choosing-the-network-interface-to-use" rel="nofollow noopener"><code>GLOO_SOCKET_IFNAME</code></a> explicitly.</p>
<pre><code class="lang-auto">getent hosts `hostname`
</code></pre>

frankdong: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> Thanks for your reply.<br>
Yes, we are using Windows Subsystem for Linux as Windows not support distributed yet. Although I see an open PR for this (but we don’t know if rpc will be support in this PR): <a href="https://github.com/pytorch/pytorch/pull/42897" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/42897</a><br>
And yes, hostname can resolve correctly. One thing that different on linux machine and WSL machine we are aware of is WSL machine share ip with host windows system but need port forwarding from windows to WSL. We already tried this solution that add port forwarding but seems rpc still can’t connect.<br>
<a href="https://github.com/microsoft/WSL/issues/4150" rel="nofollow noopener">https://github.com/microsoft/WSL/issues/4150</a></p>
<p>So question is say rank0 is listening on one port 12560 for example, is this the only port will be used on rank0 machine during rpc initialization and connection?<br>
Also, is there any way to output detailed information from rpc during initialization so we can find more details here. Ideally we will be able to know what address/port rank0 is listening on and where other ranks are connecting to, whether there is some mismatch on the address or other things cause the rpc initialization hangs.</p>

ptrblck: <p>Unfortunately, I’m not familiar with it and don’t know a specific user who might be.<br>
Maybe <a class="mention" href="/u/maxiluk">@maxiluk</a> or <a class="mention" href="/u/peterjc123">@peterjc123</a> would know more.</p>

mrshenli: <aside class="quote no-group" data-username="frankdong" data-post="3" data-topic="96429">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/frankdong/48/28780_2.png" class="avatar"> frankdong:</div>
<blockquote>
<p>Although I see an open PR for this (but we don’t know if rpc will be support in this PR):</p>
</blockquote>
</aside>
<p>Yep, MSFT experts are helping us adding Windows support to PT Distributed. The first step focuses on DDP only, but we do plan to cover all features in the distributed package in future releases.</p>
<blockquote>
<p>So question is say rank0 is listening on one port 12560 for example, is this the only port will be used on rank0 machine during rpc initialization and connection?</p>
</blockquote>
<p>I see. No, that port is only used for rendezvous during initialization. RPC backend will grab a port for each pair of workers, which are not visible to users.</p>
<p>cc <a class="mention" href="/u/lcw">@lcw</a> any suggestion on how RPC can work with port forwarding?</p>

frankdong: <aside class="quote no-group" data-username="mrshenli" data-post="5" data-topic="96429">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>RPC backend will grab a port for each pair of workers, which are not visible to users.</p>
</blockquote>
</aside>
<p>Do we know the port range so we can have a test</p>

lcw: <p>I’m not super familiar with port forwarding on WSL, but I assume it does some sort of NAT, right? In that case, I suspect it would be enough if only the listening socket(s) on each server had a fixed port which is correctly forwarded? (as then all other sockets that are accepted will have a random port but the NAT would be aware of them and would handle them)</p>
<p>Unfortunately, at this moment, also the port of the listening socket(s) is picked at random (well, not at random, but we’re letting the kernel give us back an arbitrary available port). We don’t support a way for the user to specify a port. Which I think means there’s no real way for you to set up port forwarding before launching the application.</p>