RPC with raw irecv, reduce, ... distributed primitives

username: iffiX
cooked: <p>Is it possible to use torch.distributed.rpc (since 1.4.0) with irecv, reduce, broadcast… primitives together? In my experience, it is possible to use send&amp;recv primitives after <code>init_rpc()</code> call, but I am not sure whether a blocking recv / reduce / all_gather call will interfere with rpc api.</p>
<p>If you have any experience with this, please let me know, thank you!</p>

username: mrshenli
cooked: <p>Yes, this is possible, but there is one caveat. The current implementation of <code>init_rpc()</code> sets the default process group. It actually shouldn’t do that, and we are fixing it (see <a href="https://github.com/pytorch/pytorch/issues/33583" rel="nofollow noopener">this issue</a>). With the current RPC implementation, I see two ways to implement this feature.</p>
<h3>Option 1</h3>
<p>First, call <code>init_rpc</code>() and let it set the default process group. Then use the <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.new_group" rel="nofollow noopener">new_group</a> API to create new set of process group instances, and only call irecv/reduce/broadcast on the new process group instances. This would avoid messing up states of the default process group that RPC is running on.</p>
<h3>Option 2</h3>
<p>Directly create process group using its constructor. See the test below as an example:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/f6f1384811b9cc722f650ed9ead8ee99938c009a/test/distributed/test_c10d.py#L1501-L1502" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/f6f1384811b9cc722f650ed9ead8ee99938c009a/test/distributed/test_c10d.py#L1501-L1502" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/f6f1384811b9cc722f650ed9ead8ee99938c009a/test/distributed/test_c10d.py#L1501-L1502</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="1501" style="counter-reset: li-counter 1500 ;">
<li>store = c10d.FileStore(self.file_name, self.world_size)</li>
<li>pg = c10d.ProcessGroupGloo(store, self.rank, self.world_size)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Just curious, do you mind share more details of your use case? What’s the motivation to combine RPC with collective communications? I saw some requirement for this to support combining RPC with DDP. Is this also your use case?</p>

username: iffiX
cooked: <p>Thank you for your response! Yeah, I have noticed that issue on github. I am also kind of expecting using some solution similar to your option 1, and now reassured by your answer, I am ready to use it.</p>
<p>The reason why I would like to use RPC with collective communications together is that while RPC mechanism is great for implementing point to point communication paradigm, it is not very handy when it comes to implementing algorithms where serveral processes are closely correlated and doing exactly the same things in parallel, that is the job for collective comm primitives. And it is very hard to implement on-demand calling using collective comm primitives as well.</p>
<p>An example of <strong>a part of</strong> my actual application is a distributed buffer used in IMPALA and APEX, where workers are continuously pulling models from trainers and doing rollouts. The topology of workers and trainers are preset and given by user, active service discovery is not considered. “Lost of a single worker does not affect any other peers” is an optional, but important feature.</p>
<p>I have one more question, <strong>how do you handle errors</strong> like a worker disconnects from the group? As far as I’m concerned, there is just a timeout argument in the rpc api and no explicit error handling descriptions for rpc or collective in the document. Maybe ZeroMQ is a better alternative if we are concerned about robustness?</p>

username: mrshenli
cooked: <p>Thanks for sharing more details.</p>
<blockquote>
<p>I have one more question, <strong>how do you handle errors</strong> like a worker disconnects from the group?</p>
</blockquote>
<p>We are working on elasticity support to allow workers to dynamically join and leave.</p>
<blockquote>
<p>As far as I’m concerned, there is just a timeout argument in the rpc api and no explicit error handling descriptions for rpc or collective in the document.</p>
</blockquote>
<p>For timeout, there are per-RPC timeout on master branch for <code>rpc_sync</code> and <code>rpc_async</code>. For timeout in <code>remote</code>, the following PR is add it.</p>
<aside class="onebox githubpullrequest" data-onebox-src="https://github.com/pytorch/pytorch/pull/38590">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/pull/38590" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">



    <div class="github-icon-container" title="Pull Request">
      <svg width="60" height="60" class="github-icon" viewBox="0 0 12 16" aria-hidden="true"><path fill-rule="evenodd" d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
    </div>

  <div class="github-info-container">



      <h4>
        <a href="https://github.com/pytorch/pytorch/pull/38590" target="_blank" rel="noopener nofollow ugc">Implement timeout support for RRefs</a>
      </h4>

    <div class="branches">
      <code>pytorch:gh/rohan-varma/129/base</code> ← <code>pytorch:gh/rohan-varma/129/head</code>
    </div>

      <div class="github-info">
        <div class="date">
          opened <span class="discourse-local-date" data-format="ll" data-date="2020-05-16" data-time="01:33:20" data-timezone="UTC">01:33AM - 16 May 20 UTC</span>
        </div>

        <div class="user">
          <a href="https://github.com/rohan-varma" target="_blank" rel="noopener nofollow ugc">
            <img alt="rohan-varma" src="https://avatars.githubusercontent.com/u/8039770?v=4" class="onebox-avatar-inline" width="20" height="20">
            rohan-varma
          </a>
        </div>

        <div class="lines" title="17 commits changed 23 files with 737 additions and 107 deletions">
          <a href="https://github.com/pytorch/pytorch/pull/38590/files" target="_blank" rel="noopener nofollow ugc">
            <span class="added">+737</span>
            <span class="removed">-107</span>
          </a>
        </div>
      </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Stack from [ghstack](https://github.com/ezyang/ghstack):
* **#38590 Implement ti<span class="show-more-container"><a href="https://github.com/pytorch/pytorch/pull/38590" target="_blank" rel="noopener nofollow ugc" class="show-more">…</a></span><span class="excerpt hidden">meout support for RRefs**

This PR implements timeout semantics for RRef for parity with rpc_sync and rpc_async. How it works:

- Timeout parameter is added to rpc.remote. If the rpc.remote call times out, note that the error won't be raised to the user in that call, as it is not blocking (similar to rpc_async). Instead, the timeout error will be raised the next time the RRef is used (either by pickling or to_here call).
- Error handling semantics are added to RRef to deal with the timeout errors. Previously, if there was an error creating the OwnerRRef, the callback on the local user would throw an error in a callback, resulting in an `std::terminate`. Instead of this, the error is now caught and surfaced to the user the next time the RRef is used. As part of this, we have added an `RPCErrorType` enum and defined RRef error handlers to handle the `RPCErrorrTypes` (currently just timeout and unknown)
- A timeout parameter is added to `to_here()` which gives the user control over the max amount of time it can block for.
- Before blocking, `to_here` checks if the RRef creation on the remote node has timed out, and if so, throws with that information.
- `ctx.prepareChildForFork()` which is called when the RRef is pickled (i.e. used as an arg over RPC) checks if the `rpc.remote()` call had timed out, and if so, raises that error to the user.
- Tests are added, primarily via delay injection.

Differential Revision: [D21588165](https://our.internmc.facebook.com/intern/diff/D21588165/)

**NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D21588165/)!</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<blockquote>
<p>no explicit error handling descriptions for rpc or collective in the document</p>
</blockquote>
<p>Right, currently, there is only a <code>wait</code> API on the returned <code>Future</code> object. Currently, you will need to try-except the <code>wait</code>. In the C++ implementation, there are error handling APIs on the <code>Future</code>, let us expose it to Python as well. Besides this, do you also need things like error listener for errors occurs in background RPC threads or something else?</p>
<blockquote>
<p>Maybe ZeroMQ is a better alternative if we are concerned about robustness?</p>
</blockquote>
<p>It depends on what do you need in the application. If you just need P2P communication without tensor/autograd support, then any matured RPC-like system can do. If you need autograd, or would like to do direct device-2-device copy (coming to v1.6), or would like to integrate with TorchScript to speed up training, then <code>torch.distributed.rpc</code> might be better. The latter is a relatively new project and there are still many gaps in the features, but we’d love to drive it together with the community into a better shape.</p>

username: iffiX
cooked: <p>Thank you for your detailed reply, currently, my project does not need more error in-depth error handling apart from try-excepting the <code>wait</code>, after a thorough consideration, I think rpc and collective comm are sufficient for the most necessary functions, for now. Whether using some more mature RPC frameworks really depends on my users requests, and I think it is unecessary at this moment.</p>
<p>Looking forward to more updates on elasticity enhancement. <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=9" title=":laughing:" class="emoji" alt=":laughing:"></p>

username: iffiX
cooked: <p>
According to the definition of function <code>_invoke_remote_builtin</code> and <code>_invoke_remote_python_udf</code><br>
in <code>pytorch/torch/csrc/distributed/rpc/init.cpp</code>, L557-591:
</p>

<pre><code class="lang-auto">     module.def(
      "_invoke_rpc_builtin",
      [](const WorkerInfo&amp; dst,
         const std::string&amp; opName,
         const float rpcTimeoutSeconds,
         const py::args&amp; args,
         const py::kwargs&amp; kwargs) {
        return std::make_shared&lt;jit::PythonFutureWrapper&gt;(
            pyRpcBuiltin(dst, opName, args, kwargs, rpcTimeoutSeconds));
      },
      py::call_guard&lt;py::gil_scoped_acquire&gt;());
</code></pre>

<p>


<br>
, which are called under the hood by <code>rpc.remote</code>, <code>rpc.rpc_async</code>, <code>rpc.rpc_sync</code>, is it correct to say that the <code>rf</code> parameter in <code>_invoke_rpc</code>:
</p>

<pre><code class="lang-auto">def _invoke_rpc(to, func, rpc_type, args=None, kwargs=None):
    if not callable(func):
        raise TypeError("function should be callable.")

    qualified_name = torch.jit._find_builtin(func)
    dst_worker_info = _to_worker_info(to)
    # If profiling is enabled, kick off the timer and retrieve back a
    # RecordFunction instance.
    rf = None
    ...
</code></pre>

<p>
defines timeout in <strong>seconds</strong> for each individual rpc call?
</p>
<p>Sorry I mixed up the python api code in 1.5.0 release and current master branch code. Now master branch provides timeout argument in python api:</p>
<pre><code class="lang-auto">@_require_initialized
def rpc_sync(to, func, args=None, kwargs=None, timeout=UNSET_RPC_TIMEOUT):
</code></pre>

username: mrshenli
cooked: <p>Yes, timeout arg is now available in <code>rpc_sync</code> and <code>rpc_async</code> APIs on master. Support for <code>remote</code> API will come soon.</p>