RPC agent timeout error when not doing anything

username: kmaeng
cooked: <p>I am using torch.distributed.rpc library and the RPC agent crashes if I don’t do any RPC call for 30 minutes.<br>
Is this an expected behavior of the RPC library? I think it crashes because it thinks something went wrong when there is no RPC traffic for 30 minutes; however, it times out and crashes even when there was no RPC traffic because my code was written that way!</p>
<p>Below is a code snippet which can reproduce the problematic behavior. If waiting for 30 minutes is too long to test, you can change the hardcoded value in /rpc/backend_registry.py file (detailed in the comment).</p>
<pre><code class="lang-auto">import torch.distributed.rpc as rpc
from torch.multiprocessing import Process

import os
import time

def do_nothing():
    pass

def test(rank, size):
    rpc.init_rpc("Rank"+str(rank), rank=rank, world_size=size)
    print("Rank %s rpc init" % rank)

    i = 0
    # To test easily, I changed &lt;PATH_TO_TORCH_LIB&gt;/torch/distributed/rpc/backend_registry.py.
    # Under def _process_group_init_backend_handler(),
    # I changed the below line
    # &gt;&gt; process_group_timeout = rpc_constants.DEFAULT_PROCESS_GROUP_TIMEOUT
    # (which makes the timeout 30 minutes), to somewhat shorter value, e.g.,
    # &gt;&gt; process_group_timeout = datetime.timedelta(seconds=10).
    # Otherwise, if I wait for 30 min the problem still occurs.

    ## Loop that does not do anything for a long time...
    while i &lt; 10:
        time.sleep(1)
        print("Rank %s %s sec passed..." % (rank, i))

        ## Uncommenting the below two lines makes the crash go away!
        ## I.e., generating some RPC traffic.
        #target = rank ^ 0x1
        #rpc.rpc_sync("Rank"+str(target), do_nothing)
        i += 1

    rpc.shutdown()
    print("Rank %s rpc shutdown" % rank)
    pass

if __name__ == "__main__":
    os.environ['MASTER_ADDR'] = "localhost"
    os.environ['MASTER_PORT'] = "29502"

    processes = []
    for rank in [0,1]:
        p = Process(target=test, args=(rank, 2, ))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

</code></pre>
<p>The error message is:</p>
<pre><code class="lang-auto">[E process_group_agent.cpp:664] Encountered exception in ProcessGroupAgent::listenLoop(): [/pytorch/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:84] Timed out waiting 10000ms for recv operation to complete on worker 1. This means that the RPC agent is in an unhealthy state and unusable.
</code></pre>
<p>I wonder if this is a bug, an expected behavior, or if I am using the API in an incorrect way. If it is an expected behavior, is there any workaround?</p>
<p>I am mainly experiencing this behavior because my code has a process that does not do any RPC calls, but instead calls functions under pytorch.distributed, such as distributed.all_reduce().<br>
I first tried not initializing rpc at all in those processes, but instead calling distributed.init_process_group().<br>
However, this made rpc.init_rpc() calls in other processes to hang or crash; I am suspecting that the problem happens because rpc.init_rpc() calls distributed.init_process_group() internally and somehow they don’t play well when some processes call init_process_group() via init_rpc() and others call it directly…<br>
If rpc timing out after 30 min is an expected behavior, maybe I need to find a way to make some processes to call rpc.init_rpc() and others to call distributed.init_process_group() without failure.</p>
<p>Thank you in advance.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/kmaeng">@kmaeng</a>, there are actually RPC activity for graceful RPC shutdown. See the code below. It’s basically using an RPC to prevent the idle process from exiting too early.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/9d0e935b489a33b87711b9d5d3525a7282ab89c1/torch/distributed/rpc/api.py#L133-L192" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/9d0e935b489a33b87711b9d5d3525a7282ab89c1/torch/distributed/rpc/api.py#L133-L192" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/9d0e935b489a33b87711b9d5d3525a7282ab89c1/torch/distributed/rpc/api.py#L133-L192</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="133" style="counter-reset: li-counter 132 ;">
<li>@_require_initialized</li>
<li>def _wait_all_workers():</li>
<li>    r"""</li>
<li>    Block until all local and remote RPC processes reach this method and wait</li>
<li>    for all outstanding work to complete. Every RPC process must call this</li>
<li>    method before exit to perform a graceful shutdown. This should be used to</li>
<li>    terminate the RPC framework, and there is no guarantee that the RPC</li>
<li>    framework will work after this method returns.</li>
<li>    """</li>
<li>    assert (</li>
<li>        _ALL_WORKER_NAMES is not None</li>
<li>    ), "`_ALL_WORKER_NAMES` is not initialized for `def _wait_all_workers`."</li>
<li>    leader_worker_name = sorted(_ALL_WORKER_NAMES)[0]</li>
<li>
</li><li>    self_worker_name = _get_current_rpc_agent().get_worker_info().name</li>
<li>
</li><li>    global _wait_all_workers_sequence_id</li>
<li>    with _wait_all_workers_dict_lock:</li>
<li>        sequence_id = _wait_all_workers_sequence_id</li>
<li>        _wait_all_workers_sequence_id += 1</li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/9d0e935b489a33b87711b9d5d3525a7282ab89c1/torch/distributed/rpc/api.py#L133-L192" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>To get around this, you can increase the default RPC timeout. Depending on the version you are using, you can either provide the timeout value in <a href="https://pytorch.org/docs/stable/rpc.html#torch.distributed.rpc.ProcessGroupRpcBackendOptions" rel="nofollow noopener"><code>init_rpc</code></a> (with v1.5), or directly call <a href="https://github.com/pytorch/pytorch/blob/v1.4.1/test/rpc_test.py#L1232" rel="nofollow noopener"><code>rpc._set_rpc_timeout</code></a> (with v1.4).</p>
<p>Or if you know for sure when a process can safely exit, you can use <code>shutdown(graceful=False)</code> and do the termination detection in application code.</p>
<p>Per op (rpc_sync/rpc_async/remote) timeout are coming soon.</p>

username: kmaeng
cooked: <p>Thank you for your response. I don’t fully understand your answer.<br>
Can you clarify some points?</p>
<ol>
<li>
<p>I don’t understand what you are trying to show with the code you linked. Are you saying I can use the _wait_all_workers() or a snippet of the code inside to work around the issue (I am already calling rpc.shutdown() at the end that internally calls this. It is just that the processes dies before reaching here)?<br>
My main issue is that the rpc process group agent is killed after 30 minutes of being idle. Are you suggesting I just let it die and do graceful=False so that other processes do not die while trying to shutdown rpc?</p>
</li>
<li>
<p>I am using v1.5 and tried giving timeout value to init_rpc, but did not work. I followed the code and figured the 30 min timeout only changes when I change this line:<br>
<a href="https://github.com/pytorch/pytorch/blob/91f451a5e69d2969d730744e98e059d05e63a84d/torch/distributed/rpc/backend_registry.py#L114" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/91f451a5e69d2969d730744e98e059d05e63a84d/torch/distributed/rpc/backend_registry.py#L114</a><br>
As you can see, that line and below takes the rpc_constants.DEFAULT_PROCESS_GROUP_TIMEOUT and calls dist.init_process_group(), without using the provided value by me. From my testing, that value was generating the error I am keep seeing (afterward, when calling ProcessGroupAgent, the timeout value I provided is getting passed, but not for the dist.init_process_group()).</p>
</li>
<li>
<p>In summary, is it normal to see the error message I posted if I call init_rpc and don’t use it for 30 minutes?</p>
</li>
</ol>
<p>Thank you for your help. I really appreciate it.</p>

username: mrshenli
cooked: <p>I see, sorry that I misread the original question.</p>
<blockquote>
<p>In summary, is it normal to see the error message I posted if I call init_rpc and don’t use it for 30 minutes?</p>
</blockquote>
<p>This is indeed a bug, and I think it is due to the following code, where the ProcessGroup RPC agent’s recvAnysource timed out. We should have passed rpc timeout to process group or set it to infinity in the listen loop.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/1f87f15ba3cd46fe5474f7783ff80ef06950e157/torch/csrc/distributed/rpc/process_group_agent.cpp#L710-L715">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/1f87f15ba3cd46fe5474f7783ff80ef06950e157/torch/csrc/distributed/rpc/process_group_agent.cpp#L710-L715" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/1f87f15ba3cd46fe5474f7783ff80ef06950e157/torch/csrc/distributed/rpc/process_group_agent.cpp#L710-L715" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/1f87f15ba3cd46fe5474f7783ff80ef06950e157/torch/csrc/distributed/rpc/process_group_agent.cpp#L710-L715</a></h4>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="710" style="counter-reset: li-counter 709 ;">
          <li>void ProcessGroupAgent::listenLoopInternal() {</li>
          <li>  while (rpcAgentRunning_.load()) {</li>
          <li>    // rank, tensor size, message type</li>
          <li>    std::vector&lt;torch::Tensor&gt; preamble = {torch::empty({4}, {torch::kInt64})};</li>
          <li>    auto work = pg_-&gt;recvAnysource(preamble, pg_-&gt;getRank());</li>
          <li>    {</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Thanks for flagging this, do you want to create an issue on github to track this? We will fix. Thanks!</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="kmaeng" data-post="3" data-topic="81264">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kmaeng/48/23707_2.png" class="avatar"> kmaeng:</div>
<blockquote>
<ul>
<li>As you can see, that line and below takes the rpc_constants.DEFAULT_PROCESS_GROUP_TIMEOUT and calls dist.init_process_group(), without using the provided value by me. From my testing, that value was generating the error I am keep seeing (afterward, when calling ProcessGroupAgent, the timeout value I provided is getting passed, but not for the dist.init_process_group()).</li>
</ul>
</blockquote>
</aside>
<p>Yes, exactly, we have a tracking issue related to this problem: <a href="https://github.com/pytorch/pytorch/issues/33583" class="inline-onebox" rel="noopener nofollow ugc">Avoid modifying default process group in init_rpc · Issue #33583 · pytorch/pytorch · GitHub</a></p>

username: kmaeng
cooked: <p>I submitted an issue (<a href="https://github.com/pytorch/pytorch/issues/38531" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/38531</a>), thanks!</p>