How to catch exceptions caused by rpc exactly

username: iffiX
cooked: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> How to catch exceptions <code>rpc.async</code>, <code>rpc.sync</code> and <code>rpc.remote</code> thrown in the <strong>caller</strong> under the following conditions, suppose a timeout is set globally (or per call):</p>
<ol>
<li>during execution, the target process crashes and exits, also closing down all rpc execution threads.</li>
<li>during execution, connection to the target process is closed</li>
<li>during execution, the timeout limit is reached</li>
<li>during execution, an exception is raised in the executed function</li>
</ol>
<p>Based on my experiments, my partial answer is:</p>
<ol>
<li>Not known ?</li>
<li>A <code>RuntimeError</code>, something like “peer reset”</li>
<li>An uncatchable <code>std::runtime_error</code>, something like:</li>
</ol>
<pre><code class="lang-auto">terminate called after throwing an instance of 'std::runtime_error'
  what():  RPC ran for more than 5000 milliseconds and timed out.
</code></pre>
<ol start="4">
<li>the exception thrown by the function, not the original exception, but wrapped in a udf exception and reraised on the caller side.</li>
</ol>
<p>The third one troubles me the most because <code>std::runtime_error</code> will cause an ugly <code>Fatal Python Error</code>:</p>
<pre><code class="lang-auto">Fatal Python error: Aborted

Thread 0x00007f916abab700 (most recent call first):
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/distributed/world.py", line 63 in _rpc_call_remote_method
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/torch/distributed/rpc/internal.py", line 153 in _run_function

Thread 0x00007f91693a8700 (most recent call first):
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/distributed/world.py", line 75 in _rpc_get_remote_paired_value
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/torch/distributed/rpc/internal.py", line 153 in _run_function

Thread 0x00007f9163fff700 (most recent call first):
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/distributed/world.py", line 75 in _rpc_get_remote_paired_value
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/torch/distributed/rpc/internal.py", line 153 in _run_function

Thread 0x00007f91527fc700 (most recent call first):
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/torch/distributed/rpc/api.py", line 554 in rpc_sync
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/torch/distributed/rpc/api.py", line 77 in wrapper
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/distributed/world.py", line 756 in _rpc_paired_class_call
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/distributed/world.py", line 597 in rpc_paired_class_sync
  File "/home/Administrator/iffi/Projects/machin/test/parallel/distributed/test_world.py", line 97 in main
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/distributed/world.py", line 46 in _exec_role
  File "/usr/lib/python3.5/threading.py", line 862 in run
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/thread.py", line 47 in run
  File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
  File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap

Thread 0x00007f9152ffd700 (most recent call first):
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/distributed/election.py", line 423 in _task_timeout
  File "/usr/lib/python3.5/threading.py", line 862 in run
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/thread.py", line 47 in run
  File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
  File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap

Thread 0x00007f91537fe700 (most recent call first):
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/distributed/election.py", line 435 in _task_keep_alive
  File "/usr/lib/python3.5/threading.py", line 862 in run
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/thread.py", line 47 in run
  File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
  File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap

Thread 0x00007f9153fff700 (most recent call first):
  File "/usr/lib/python3.5/threading.py", line 297 in wait
  File "/usr/lib/python3.5/queue.py", line 173 in get
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/distributed/election.py", line 491 in _task_handle
  File "/usr/lib/python3.5/threading.py", line 862 in run
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/thread.py", line 47 in run
  File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
  File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap

Thread 0x00007f9160ff9700 (most recent call first):
  File "/usr/lib/python3.5/threading.py", line 293 in wait
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/event.py", line 66 in wait
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/distributed/role_dispatcher.py", line 234 in _task_dispatch
  File "/usr/lib/python3.5/threading.py", line 862 in run
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/thread.py", line 47 in run
  File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
  File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap

Thread 0x00007f91617fa700 (most recent call first):
  File "/usr/lib/python3.5/threading.py", line 293 in wait
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/event.py", line 66 in wait
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/distributed/world.py", line 302 in _task_run_dispatched_roles
  File "/usr/lib/python3.5/threading.py", line 862 in run
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/thread.py", line 47 in run
  File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
  File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap

Thread 0x00007f91e4362700 (most recent call first):
  File "/home/Administrator/iffi/Projects/machin/test/parallel/distributed/test_world.py", line 145 in subproc_start_world_with_roles
  File "/home/Administrator/iffi/Projects/machin/test/parallel/util_run_multi.py", line 16 in process_main
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93 in run
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/process.py", line 52 in run
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249 in _bootstrap
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 74 in _launch
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 20 in __init__
  File "/usr/lib/python3.5/multiprocessing/context.py", line 267 in _Popen
  File "/home/Administrator/iffi/Projects/machin/machin/parallel/process.py", line 25 in _Popen
  File "/usr/lib/python3.5/multiprocessing/process.py", line 105 in start
  File "/home/Administrator/iffi/Projects/machin/test/parallel/util_run_multi.py", line 27 in processes
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/fixtures.py", line 788 in call_fixture_func
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/fixtures.py", line 964 in pytest_fixture_setup
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/manager.py", line 87 in &lt;lambda&gt;
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/fixtures.py", line 914 in execute
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/fixtures.py", line 584 in _compute_fixture_value
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/fixtures.py", line 503 in _get_active_fixturedef
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/fixtures.py", line 487 in getfixturevalue
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/fixtures.py", line 477 in _fillfixtures
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/fixtures.py", line 297 in fillfixtures
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/python.py", line 1483 in setup
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/runner.py", line 373 in prepare
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/runner.py", line 123 in pytest_runtest_setup
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/manager.py", line 87 in &lt;lambda&gt;
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/runner.py", line 217 in &lt;lambda&gt;
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/runner.py", line 244 in from_call
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/runner.py", line 217 in call_runtest_hook
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/runner.py", line 186 in call_and_report
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/runner.py", line 94 in runtestprotocol
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/runner.py", line 85 in pytest_runtest_protocol
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/manager.py", line 87 in &lt;lambda&gt;
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/main.py", line 272 in pytest_runtestloop
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/manager.py", line 87 in &lt;lambda&gt;
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/main.py", line 247 in _main
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/main.py", line 191 in wrap_session
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/manager.py", line 87 in &lt;lambda&gt;
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/Administrator/iffi/Projects/machin/venv/lib/python3.5/site-packages/_pytest/config/__init__.py", line 125 in main
  File "/data/software/pycharm/pycharm-2020.1.2/plugins/python/helpers/pycharm/_jb_pytest_runner.py", line 43 in &lt;module&gt;
</code></pre>
<p>Is there any clean way to deal with the first three conditions? The fourth one is simple. And why pybind11 is not converting the third <code>std::runtime_error</code> to a catchable python <code>RuntimeError</code> ?</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/iffix">@iffiX</a></p>
<blockquote>
<ol>
<li>during execution, the target process crashes and exits, also closing down all rpc execution threads.</li>
<li>during execution, connection to the target process is closed</li>
</ol>
</blockquote>
<p>Prior to v1.6, ProcessGroup is the only available backend, which requires all processes to be alive. So RPC gang cannot survive these failures. Even if you can catch it in application code, it will leave subsequent RPC behaviors in an undefined state, unless there is a global recovery process (we do have plans to providing this).</p>
<p>We will introduce <a href="https://github.com/pytorch/tensorpipe" rel="noopener nofollow ugc">TensorPipe</a> backend for RPC in v1.6, which is a P2P comm library. But in the first experimental version, it still has some part depends on ProcessGroup, so I think it still wouldn’t tolerate such failures in v1.6. There is also an ongoing project to provide elasticity to RPC.</p>
<p>cc <a class="mention" href="/u/lcw">@lcw</a> <a class="mention" href="/u/agolynski">@agolynski</a></p>
<blockquote>
<ol start="3">
<li>during execution, the timeout limit is reached</li>
</ol>
</blockquote>
<p>This should thrown a <code>RuntimeError</code> for the timeout. And try-except on <code>RuntimeError</code> type should be able to catch it. See the code below:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/testing/_internal/distributed/rpc/rpc_test.py#L2501-L2502">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/testing/_internal/distributed/rpc/rpc_test.py#L2501-L2502" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/testing/_internal/distributed/rpc/rpc_test.py#L2501-L2502" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/testing/_internal/distributed/rpc/rpc_test.py#L2501-L2502</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="2501" style="counter-reset: li-counter 2500 ;">
          <li>with self.assertRaisesRegex(RuntimeError, error_str):</li>
          <li>    fut.wait()</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>If this is not so in your application, then it is a bug that we need to fix. Please let us know how we can reproduce it. Or if you have other suggestions on how we should report timeout error, please also let us know your suggestions.</p>
<p>cc <a class="mention" href="/u/rvarm1">@rvarm1</a></p>
<blockquote>
<ol start="4">
<li>during execution, an exception is raised in the executed function</li>
</ol>
</blockquote>
<p>If there is an exception during remote execution, this should be thrown on the caller side. See the code below:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/testing/_internal/distributed/rpc/rpc_test.py#L1625-L1631">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/testing/_internal/distributed/rpc/rpc_test.py#L1625-L1631" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/testing/_internal/distributed/rpc/rpc_test.py#L1625-L1631" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/testing/_internal/distributed/rpc/rpc_test.py#L1625-L1631</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="1625" style="counter-reset: li-counter 1624 ;">
          <li>@dist_init</li>
          <li>def test_py_raise_in_user_func(self):</li>
          <li>    n = self.rank + 1</li>
          <li>    dst_rank = n % self.world_size</li>
          <li>    fut = rpc.rpc_async(worker_name(dst_rank), raise_func)</li>
          <li>    with self.assertRaises(ValueError):</li>
          <li>        fut.wait()</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>This is how the Python error is captured:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/distributed/rpc/internal.py#L152-L171">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/distributed/rpc/internal.py#L152-L171" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/distributed/rpc/internal.py#L152-L171" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/distributed/rpc/internal.py#L152-L171</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="152" style="counter-reset: li-counter 151 ;">
          <li>def _run_function(python_udf):</li>
          <li>    r"""</li>
          <li>    This function is exclusively called from C++.</li>
          <li>    See ``torch/csrc/distributed/rpc/python_rpc_handler.cpp``.</li>
          <li></li>
          <li>    Runs a Python UDF and returns its return value.</li>
          <li>    Wraps any exception in ``RemoteException`` if the function raises.</li>
          <li>    """</li>
          <li>    try:</li>
          <li>        if isinstance(python_udf, AttributeError):</li>
          <li>            raise python_udf</li>
          <li>        result = python_udf.func(*python_udf.args, **python_udf.kwargs)</li>
          <li>    except Exception as e:</li>
          <li>        # except str = exception info + traceback string</li>
          <li>        except_str = (</li>
          <li>            f"On {_get_current_rpc_agent().get_worker_info()}:\n"</li>
          <li>            f"{repr(e)}\n{traceback.format_exc()}"</li>
          <li>        )</li>
          <li>        result = RemoteException(except_str, type(e))</li>
          <li>    return result</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>This is how non-user error is captured:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/csrc/distributed/rpc/request_callback_impl.cpp#L907-L921">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/csrc/distributed/rpc/request_callback_impl.cpp#L907-L921" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/csrc/distributed/rpc/request_callback_impl.cpp#L907-L921" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/67a4f375cdf06113ca959b4e16739edb666f243f/torch/csrc/distributed/rpc/request_callback_impl.cpp#L907-L921</a></h4>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="907" style="counter-reset: li-counter 906 ;">
          <li>try {</li>
          <li>  processRpc(*rpc, messageType, id, retFuture);</li>
          <li>} catch (py::error_already_set&amp; e) {</li>
          <li>  retFuture-&gt;markCompleted(handleError(e, messageType, id));</li>
          <li>  // There are request callback impls in Python, where Python</li>
          <li>  // exceptions could be thrown. For releasing Python exception</li>
          <li>  // py::objects, GIL must be held.</li>
          <li>  py::gil_scoped_acquire acquire;</li>
          <li>  e.restore(); // Release ownership on py::objects and also restore</li>
          <li>               // Python Error Indicator.</li>
          <li>  PyErr_Clear(); // Clear the Python Error Indicator as we has</li>
          <li>                 // recorded the exception in the response message.</li>
          <li>} catch (std::exception&amp; e) {</li>
          <li>  retFuture-&gt;markCompleted(handleError(e, messageType, id));</li>
          <li>}</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Please let us know if we missed any. Thanks!</p>

username: mrshenli
cooked: <p>BTW, which version of PyTorch are you using currently?</p>

username: iffiX
cooked: <p>My local development version is 1.5.0, my test version is your latest docker build.</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"> It’s really sad to hear that currently pytorch rpc cannot handle the 1st and 2nd condition, since that’s what my application code is designed to do. I will try to repoduce the 3rd condition with simpler code, but that might be very difficult since currently there is no way to log all events just before the “fatal abort” happens.</p>
<p>Anyway, thanks for your response! I still want to ask: what kind of P2P mechanism are you going to provide in the TensorPipe backend? could you please clarify your and your collegues’ plans a little bit more?</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="iffiX" data-post="4" data-topic="88856">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/48/24443_2.png" class="avatar"> iffiX:</div>
<blockquote>
<p>It’s really sad to hear that currently pytorch rpc cannot handle the 1st and 2nd condition, since that’s what my application code is designed to do.</p>
</blockquote>
</aside>
<p>Could you please elaborate more on the requirement? We understand it is important to provide failure-recovery + elasticity, but haven’t seen people explicitly requesting this yet, so that didn’t get into our top-priority in past releases. If this is a commonly required feature that might block many use cases, we will reprioritize work items and try to get this done sooner.</p>
<blockquote>
<p>Anyway, thanks for your response! I still want to ask: what kind of P2P mechanism are you going to provide in the TensorPipe backend? could you please clarify your and your collegues’ plans a little bit more?</p>
</blockquote>
<p>This will be released in v1.6 very soon. TensorPipe no longer requires rendezvous or full participation from all processes. So technically, crashed processes do not prevent the rest processes to function correctly (not yet so in v1.6). And it should also be faster than ProcessGroup-based RPC backend. Eventually, we will retire the ProcessGroup-based RPC backend, and make TensorPipe RPC backend as the default option.</p>
<p>cc the main contributor of TensorPipe <a class="mention" href="/u/lcw">@lcw</a>   <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: iffiX
cooked: <p>Actually its just an experimental failure tolerance feature, when I implemented feature, I was expecting your rpc layer to be similar to a normal unreliable connection, that is, a lossy link between 2 processes will not affect any other processes, if it fail, it just fail silently between these 2 processes only, and throws a detectable error. It doesn’t have to be recoverable. Elasticity is also not required, currently it only deals with a preset number of processes, suppose you have 100 work “roles”, you may run it with 1 process or 200 processes, the system is fully distributed and can tollerate losses, it will reschedule the failed role to other healthy processes.</p>
<p>It’s not an imporatant core feature, although I spent quite some time to implement and test it. I have removed it from my framework today, for now <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"></p>
<p>I have read descriptions of the TensorPipe, its great to have a smart backend which can choose the best way to move data around transprently, looking forward to it. <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>

username: mrshenli
cooked: <p>Sorry about the inconvenience!</p>
<aside class="quote no-group" data-username="iffiX" data-post="6" data-topic="88856">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/48/24443_2.png" class="avatar"> iffiX:</div>
<blockquote>
<p>a lossy link between 2 processes will not affect any other processes, if it fail, it just fail silently between these 2 processes only, and throws a detectable error. It doesn’t have to be recoverable. Elasticity is also not required, currently it only deals with a preset number of processes</p>
</blockquote>
</aside>
<p>I see. In this case, looks like we just need to:</p>
<ol>
<li>remove the ProcessGroup usage from TensorPipe RPC agent</li>
<li>let TensorPipe throw proper errors.</li>
</ol>
<p><a class="mention" href="/u/lcw">@lcw</a> did I miss anything?</p>

username: iffiX
cooked: <p>If that is the case, I can try to restructure my code and submit this feature as a pull request to your distributed module, if you think this is valuable for your rpc module.</p>
<p>The feature is structured in the following way:</p>
<pre><code class="lang-auto">rpc layer &lt;-&gt; election layer -&gt; role dispatcher layer &lt;-&gt; role runner &lt;-&gt; wrapped highlevel apis
</code></pre>
<p>High level apis works like:</p>
<pre><code class="lang-auto">WORLD                               (only initialize once, tell it what roles you want to run)
\-&gt; create_collective_group(...)    (create a sub group which supports irecv, isend, all_gather etc.)
\-&gt; create_rpc_group(name, roles)   (create a rpc group)
\-&gt; get_rpc_group(name, role)       (find a group handle created by a local role or remote role)

CollectiveGroup
\-&gt; send
\-&gt; recv
\-&gt; isend
\-&gt; irecv
...
\-&gt; barrier

RpcGroup
\-&gt; rpc_pair(key, value)           (pair a value to the group, so that it can be accessed, locally or remotely)
\-&gt; rpc_get_paired(key)            (get a paired value in this group)
\-&gt; rpc_sync
\-&gt; rpc_async
\-&gt; rpc_remote
\-&gt; rpc_paired_class_sync          (invoke a method on the paired class instance)
\-&gt; rpc_paired_class_async
\-&gt; rpc_paired_class_remote
\-&gt; rpc_paired_model_sync          (perform a forward operation on the registered model)
\-&gt; rpc_paired_model_async
\-&gt; rpc_paired_model_remote   
</code></pre>
<p>Users may start a role, register a service on this role, and access this service from another role like:</p>
<pre><code class="lang-auto">class WorkerService(object):
    # An example stateful service class
    _count = 0

    def counter(self):    
        self._count += 1
        return self._count


class Worker(RoleBase):
    def __init__(self, index):
        super(Worker, self).__init__(index)
        self.service = WorkerService()
        self.group = get_world().create_rpc_group("Employees", roles=[
            "Worker:0", "Worker:1", "Worker:2"
        ])
        # expose service
        self.group.rpc_pair("worker_service", self.service)

    def main(self):
        while True:
            pass

class Manager(RoleBase):
    def __init__(self, index):
        super(Manager, self).__init__(index)
        self.group = get_world().get_rpc_group("Employee", "Worker:{}".format(index))

    def main(self):
       for i in range(10):
            self.group.rpc_paired_class_sync(
                to="Worker:{}".format(self.role_index),
                cls_method=WorkerService.counter,
                name="worker_service",
            )

if __name__ == "__main__":
    # suppose process rank is `rank`
    # suppose there are 3 processes
    # start 3 managers and 3 workers
    world = World(world_size=3, rank=rank, 
                  roles={"Worker": (Worker, 3),
                         "Manager": (Manager, 3)},
                  rpc_timeout=0.5, election_timeout=0.3, logging=True)
</code></pre>
<p>What do you think about this design?</p>

username: mrshenli
cooked: <p>Thanks a lot for this proposal!!</p>
<p>There was a proposal for the role-based RPC design with some similar concepts from <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a>, we might be able to join forces there.</p>
<p><a class="mention" href="/u/iffix">@iffiX</a> Regarding the PR submission, we are grateful about the intent for contributing! Before making decisions on the commitment, may I ask how much bandwidth will you be able to allocate to this project? Since this can be a major new feature, most likely we will need to go through the formal design proposal review --&gt; API review --&gt; code review process and will also need examples (in <a href="https://github.com/pytorch/examples" rel="nofollow noopener">pytorch/examples</a> repo) and tutorials (in <a href="https://github.com/pytorch/tutorials" rel="nofollow noopener">pytorch/tutorials</a> repo) for it. This can become a very involved effort, so that we will need to make sure we have all the things we need to hit the finishing line.</p>

username: iffiX
cooked: <p>Hmm, I can do that, I will finish testing the rest of my framework first, might take a week or so, then pour in major effort to complete this, currently I am studing &amp; working at home full time. I am really proud to hear some affirmation!</p>
<p>So is there any document for the whole process of the proposal? It would be much easier if there is any template to refer to.</p>

username: iffiX
cooked: <p>BTW, I would also like to see the proposal of your role-based RPC design from Kiuk_Chung <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>

username: mrshenli
cooked: <p>Awesome!! And yes, let’s wait for comments from <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a></p>
<p>If this looks OK, we can then start from publishing an RFC design issue. Here are some examples:</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/23110" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/23110" target="_blank" rel="nofollow noopener">[RFC] RPC Based Distributed Model Parallel</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-07-19" data-time="21:33:47" data-timezone="UTC">09:33PM - 19 Jul 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/mrshenli" target="_blank" rel="nofollow noopener">
          <img alt="mrshenli" src="https://avatars2.githubusercontent.com/u/16999635?v=4" class="onebox-avatar-inline" width="20" height="20">
          mrshenli
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">with @pritamdamania87 @zhaojuanmao @aazzolini @gqchen @pietern @satgera @ezyang @zdevito @suo @manojkris @gchanan @soumith @dzhulgakov @yifuwang @bddppq @joxu-cn @dwarakrajagopal @jspisak
PyTorch currently provides...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">feature</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: rpc</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/38174" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/38174" target="_blank" rel="nofollow noopener">[RFC] Join-based API to support uneven inputs in DDP</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-05-09" data-time="02:00:25" data-timezone="UTC">02:00AM - 09 May 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/rohan-varma" target="_blank" rel="nofollow noopener">
          <img alt="rohan-varma" src="https://avatars2.githubusercontent.com/u/8039770?v=4" class="onebox-avatar-inline" width="20" height="20">
          rohan-varma
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🚀 Feature
with @pritamdamania87 @mrshenli @zhaojuanmao
This RFC is to summarize the current proposal for supporting uneven inputs across different DDP processes. Related...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">feature</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: distributed</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/36071" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/36071" target="_blank" rel="nofollow noopener">[RFC] Async User Function for RPC</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-04-06" data-time="16:04:04" data-timezone="UTC">04:04PM - 06 Apr 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/mrshenli" target="_blank" rel="nofollow noopener">
          <img alt="mrshenli" src="https://avatars2.githubusercontent.com/u/16999635?v=4" class="onebox-avatar-inline" width="20" height="20">
          mrshenli
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Impact
Currently, every RPC request occupies an RPC thread on the server side until done. However, if there are nested RPC calls...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">feature</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: rpc</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/39272" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/39272" target="_blank" rel="nofollow noopener">[RFC] DDP Communication Hook</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-05-29" data-time="21:10:01" data-timezone="UTC">09:10PM - 29 May 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/pritamdamania87" target="_blank" rel="nofollow noopener">
          <img alt="pritamdamania87" src="https://avatars1.githubusercontent.com/u/9958665?v=4" class="onebox-avatar-inline" width="20" height="20">
          pritamdamania87
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Motivation
There are several algorithms like GossipGrad and gradient compression which involve different communication strategies for parameter syncs while running Distributed DataParallel...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">enhancement</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: distributed</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>This will draw people into the discussion.</p>

username: Kiuk_Chung
cooked: <p><a class="mention" href="/u/iffix">@iffiX</a>, <a class="mention" href="/u/mrshenli">@mrshenli</a> I’ll publish the <code>torch.distributed.app</code> proposal that we’ve reviewed internally as a RFC on github shortly - just need to format it in markdown style.</p>

username: Kiuk_Chung
cooked: <p><a class="mention" href="/u/iffix">@iffiX</a> – here it is: <a href="https://github.com/pytorch/pytorch/issues/41425" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/41425</a>. Looking forward to collaborating on this!</p>

username: lcw
cooked: <p><a class="mention" href="/u/iffix">@iffiX</a> A lot of interesting stuff here, let me get to it in order.</p>
<p>Your points <span class="hashtag">#1</span> (failure of a node) and <span class="hashtag">#2</span> (failure of a link) are indeed handled poorly by the ProcessGroup backend as they bring all other nodes and links down with them. The TensorPipe backend is more resilient and only fails the affected requests. At this stage, a failed link will remain failed forever (although we’re discussing how to improve this in <a href="https://github.com/pytorch/pytorch/issues/40936" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/40936</a>, please chime in there). A failed node is harder to deal with, as nodes can be stateful and thus restarting it doesn’t mean that the other nodes can resume talking to it as if nothing had happened. The current approach chosen by Elastic is to restart all nodes when one fails. That’s drastic but safe. Also, even in the TensorPipe backend, we currently have nodes exchange their addresses at the beginning, so if a node (re-)joins later it won’t be able to make its address known to the other ones.</p>
<p>The other point where the TensorPipe backend performs a collective operation among all nodes is during <em>graceful</em> shutdown. It is an intrinsic requirement of the API that this method operates like a barrier. Currently that is implemented using process groups and thus suffers from their problems, but even if we reimplemented it on top of TensorPipe it wouldn’t be much different: a failed link/node would cause all others to fail too. I don’t see a way around it while still upholding that requirement. However, if you have your own logic on the nodes’ lifetimes, you probably can do an <em>ungraceful</em> shutdown and thus avoid this whole problem entirely: in that case the TensorPipe backend never uses its process group so it doesn’t matter if it fails.</p>
<p>As for your point <span class="hashtag">#3</span>, to me it looks like we’re raising an exception in a function marked as <code>noexcept</code>. If that’s the case it should be an easy fix, but it would greatly help if you could provide a C++ stack trace. Just attach GDB to the process, set up exception catching (<code>catch throw</code>) and then let the process resume as usual (<code>continue</code>). When GDB then catches an exception you can get the stack trace through <code>backtrace</code>. There may be harmless exceptions that are fired before that last fatal one, so please make sure you get the right one. It would also help if your build has debug symbols, I don’t know if the default PyTorch one does.</p>
<p>Point <span class="hashtag">#4</span> is particularly tricky, and also relates to the ones above. Dealing with exceptions is hard because, in principle, the RPC module is available to both C++ and Python, which have different exception types and thus it’s hard to preserve and convert between them. Moreover, the future objects have an error state which only contains a string message (see <a href="https://github.com/pytorch/pytorch/blob/86a2bdc35e03fbe4d66fa2666b49efc4f84855fc/torch/csrc/utils/future.h#L11-L23" rel="nofollow noopener">here</a>), so there is no way to store a “error type” in there. The way the RPC module currently propagates the type from the remote UDF to the client is a bit hacky: it involves marking the future as successful, with a result value that contains the exception type and message, and then use an “unwrap” function that fires when accessing that value that raises it instead of returning it (see <a href="https://github.com/pytorch/pytorch/blob/86a2bdc35e03fbe4d66fa2666b49efc4f84855fc/torch/csrc/distributed/rpc/init.cpp#L624-L636" rel="nofollow noopener">here</a>). So it means that an errored UDF will return a successful future, but from Python there’s no way to distinguish them. A real failed future (which you get from example in case of an I/O error) will instead always raise a RuntimeError just because it has no way of specifying anything else.</p>
<p>All these problems are in principle solvable (storing a Python exception in C++ as pybind11::error_already_set, converting a C++ exception to Python using pybind11), but this requires changes to the future class which is a top-level util of PyTorch used also by other modules (like JIT) and that makes it hard to change it.</p>

username: iffiX
cooked: <p>I think dropping all processes if one connection has failed is pretty costly, but acceptable under most conditions.<br>
I will try to collect the stack trace for point <span class="hashtag">#3</span> once I got time.</p>