Multiple nodes with Pytorch (Only CPUs)

ph0123: <p>Hi,</p>
<p>For single node, I set</p>
<pre><code class="lang-auto">os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '29500'
</code></pre>
<p>and the size is as input parameter.</p>
<p>However, with multiple nodes, we have to set differently. But I did now know how to set it?</p>
<p>For example, I know the node names with 4 nodes as below.</p>
<pre><code class="lang-auto">C1-01
C1-02
C2-01
C2-02
</code></pre>
<p>When I submit the job, the node names will change.<br>
How to set MASTER_ADDR for the program?</p>
<p>Thanks,</p>

mrshenli: <p>Hey <a class="mention" href="/u/ph0123">@ph0123</a> do you mean before submitting jobs, neither node name nor node IP addresses are known to you?</p>

ph0123: <p>Hi,</p>
<p>I submit the batch file. In the batch file, I can get the node names of all nodes, which assigned by the server.<br>
For each submission, the node names will be changed.</p>
<p>I also have another question. When I run with real data (big data), the gloo backend is stopped after 60 seconds in my program.<br>
How to set up the time out for this situation. The program will run with 20-30 minutes.</p>
<p>Thanks,</p>

mrshenli: <aside class="quote no-group" data-username="ph0123" data-post="3" data-topic="84865">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ph0123/48/22888_2.png" class="avatar"> ph0123:</div>
<blockquote>
<p>I submit the batch file. In the batch file, I can get the node names of all nodes, which assigned by the server.<br>
For each submission, the node names will be changed.</p>
</blockquote>
</aside>
<p>In that case, I wonder if it would be possible to programmably figure out the master. E.g., ask all processes to sort all node names and then always use the first one?</p>
<blockquote>
<p>How to set up the time out for this situation. The program will run with 20-30 minutes.</p>
</blockquote>
<p>Are you using RPC or DDP? For RPC, checkout this PR (<a href="https://github.com/pytorch/pytorch/pull/38577" class="inline-onebox" rel="noopener nofollow ugc">Larger timeout for operations against ProcessGroup for RPC by rohan-varma · Pull Request #38577 · pytorch/pytorch · GitHub</a>). For DDP, <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group" rel="noopener nofollow ugc"><code>init_process_group</code></a> does take a timeout argument.</p>

mrshenli: <p>The 60 second  timeout might be the default RPC timeout. Which version of PyTorch are you using?</p>
<p>In v1.4, there is a hidden <a href="https://github.com/pytorch/pytorch/blob/v1.4.0/test/rpc_test.py#L1232" rel="nofollow noopener"><code>_set_rpc_timeout</code></a> API.</p>
<p>In v1.5, you can customize the <a href="https://pytorch.org/docs/stable/rpc.html#torch.distributed.rpc.ProcessGroupRpcBackendOptions" rel="nofollow noopener"><code>ProcessGroupRpcBackendOptions</code></a> to provide a timeout.</p>

mrshenli: <p>We will be adding per-RPC timeout in v1.6.  See</p>
<ol>
<li><a href="https://github.com/pytorch/pytorch/issues/32686" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/32686</a></li>
<li><a href="https://github.com/pytorch/pytorch/issues/36000" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/36000</a></li>
</ol>

ph0123: <p>Thanks,<br>
For multiple nodes, I think I can print out to a file. The first name node is the master node.<br>
Let me try the timeout.</p>

ph0123: <p>HI,<br>
I try to set the time out as in the git. <a href="https://github.com/pytorch/pytorch/issues/32686" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/32686</a></p>
<pre><code class="lang-auto">rpc.rpc_async(my_target, add_outlinks, args=(arr_send[i],source),timeout=None)
</code></pre>
<p>The error:</p>
<pre><code class="lang-auto">TypeError: rpc_async() got an unexpected keyword argument 'timeout'
</code></pre>
<p>Perhaps, rpc is not have the “timeout” parameter.</p>
<p>Thanks,</p>

mrshenli: <p>That per-RPC timeout is only for v1.6 which hasn’t been released yet. The feature available on master now though, but you will need to either use nightly binary or build from source to get that feature.  If you are using v1.4 or v1.5, please try the other two options mentioned above.</p>
<p>What does the following code print in your environment?</p>
<pre><code class="lang-python">import torch
torch.__version__
</code></pre>

ph0123: <p>I used pytorch v1.5<br>
Thanks,</p>

mrshenli: <p>Cool, then below should be the way to go. The link below points to the doc that contains an example.</p>
<blockquote>
<p>In v1.5, you can customize the <a href="https://pytorch.org/docs/stable/rpc.html#torch.distributed.rpc.ProcessGroupRpcBackendOptions" rel="noopener nofollow ugc"> <code>ProcessGroupRpcBackendOptions</code> </a> to provide a timeout.</p>
</blockquote>

ph0123: <p>Yes,</p>
<pre><code class="lang-auto">rpc.init_rpc(my_name, rank=rank, world_size=size, rpc_backend_options=rpc.ProcessGroupRpcBackendOptions(num_send_recv_threads=16,datetime.timedelta(seconds=1000)))  # initial_rpc

</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">    rpc.init_rpc(my_name, rank=rank, world_size=size, rpc_backend_options=rpc.ProcessGroupRpcBackendOptions(num_send_recv_threads=16,datetime.timedelta(seconds=1000)))  # initial_rpc
                                                                                                                                    ^
SyntaxError: positional argument follows keyword argument
</code></pre>

mrshenli: <p>Thanks for reporting, it’s an error in the doc, I think it needs to be:</p>
<pre><code class="lang-python">rpc.ProcessGroupRpcBackendOptions(
    num_send_recv_threads=16,
    rpc_timeout=datetime.timedelta(seconds=1000)
)
</code></pre>
<p>Let me try.</p>

mrshenli: <pre><code class="lang-python">import datetime, os
from torch.distributed import rpc

os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '29500'

rpc.init_rpc(
    "worker",
    rank=0,
    world_size=1,
    rpc_backend_options=rpc.ProcessGroupRpcBackendOptions(
        num_send_recv_threads=16,
        rpc_timeout = datetime.timedelta(seconds=1000)  # note that this will change to float type to support TorchScript integration. 
    )
)

rpc.shutdown()
</code></pre>
<p>Yep, above should work. Will submit a PR to fix.</p>

mrshenli: <aside class="onebox githubpullrequest">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/pull/39787" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Pull Request">
    <svg width="60" height="60" class="github-icon" viewBox="0 0 12 16" aria-hidden="true"><path d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/pull/39787" target="_blank" rel="nofollow noopener">[Easy Review] Fix ProcessGroupRpcBackendOptions Doc</a>
    </h4>

    <div class="branches">
      <code>pytorch:gh/mrshenli/193/base</code> ← <code>pytorch:gh/mrshenli/193/head</code>
    </div>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-06-10" data-time="16:12:46" data-timezone="UTC">04:12PM - 10 Jun 20 UTC</span>
      </div>

      <div class="user">
        <a href="https://github.com/mrshenli" target="_blank" rel="nofollow noopener">
          <img alt="mrshenli" src="https://avatars2.githubusercontent.com/u/16999635?v=4" class="onebox-avatar-inline" width="20" height="20">
          mrshenli
        </a>
      </div>

      <div class="lines" title="1 commits changed 1 files with 1 additions and 1 deletions">
        <a href="https://github.com/pytorch/pytorch/pull/39787/files" target="_blank" rel="nofollow noopener">
          <span class="added">+1</span>
          <span class="removed">-1</span>
        </a>
      </div>
    </div>

  </div>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>