Pytorch RPC maximum number of concurrent RPCs?

jeremysalwen: <p>(Running on the latest pytorch nightly)</p>
<p>I am attempting to implement distributed RL training setup with batched inference (similar to <a href="https://pytorch.org/tutorials/intermediate/rpc_async_execution.html" class="inline-onebox" rel="noopener nofollow ugc">Implementing Batch RPC Processing Using Asynchronous Executions — PyTorch Tutorials 1.8.0 documentation</a>).  I have working setup, with a small number of RPCs per process (12 processes, with 15 “play_game” RPCs per process active at once).</p>
<p>However, when I attempt to increase the number of games played simultaneously by the worker processes (from 15 to 16 RPCs), instead it freezes, eventually outputting the error</p>
<pre><code>[E thread_pool.cpp:112] Exception in thread pool task: The RPC has not succeeded after the specified number of max retries (5).
</code></pre>
<p>hundreds of times after several minutes.</p>
<p>The strange thing is that 15 RPCs per process consistently succeeds, while 16 RPCs per process consistently fails.  Is this a limit on the number of RPCs that can be in flight?</p>
<p>The test I am running is available at <a href="https://github.com/JDBumgardner/stone_ground_hearth_battles/blob/master/tests/test_pytorch_distributed.py" class="inline-onebox" rel="noopener nofollow ugc">stone_ground_hearth_battles/test_pytorch_distributed.py at master · JDBumgardner/stone_ground_hearth_battles · GitHub</a></p>

mrshenli: <p>Hey <a class="mention" href="/u/jeremysalwen">@jeremysalwen</a>, does increase the value of <code>num_worker_threads</code> help? <a href="https://pytorch.org/docs/master/rpc.html#torch.distributed.rpc.TensorPipeRpcBackendOptions" class="inline-onebox" rel="noopener nofollow ugc">Distributed RPC Framework — PyTorch master documentation</a></p>
<p>Its default value is 16.</p>

jeremysalwen: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> Yes, that seems to fix the issue.  However, I am still puzzled about the original behavior.  Running out of worker threads should cause the execution of the RPC to be blocked until the threads are freed, but it seems like instead it is breaking other things.</p>
<p>This sounds like a bug, no?</p>
<p>To simplify my setup, I basically have two processes, process A and process B.  Process A is sending 16 RPCs to process B, and all these RPCs are completely independent of each other.  Each RPC from A-&gt;B internally sends a sequence of RPCs back to A.  I don’t see why this should cause a deadlock, even if the thread pool in each process is size 1, because only 1 simultaneous RPC per process is required to make forward progress.</p>
<p>My guess is that the thread pool is being reused to both send and receive RPCs, so all 16 threads are taken by the RPCs from A to B, but now all these threads are stuck, because they are unable to make RPCs from B to A?</p>

mrshenli: <aside class="quote no-group" data-username="jeremysalwen" data-post="3" data-topic="115550">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jeremysalwen/48/30510_2.png" class="avatar"> jeremysalwen:</div>
<blockquote>
<p>My guess is that the thread pool is being reused to both send and receive RPCs, so all 16 threads are taken by the RPCs from A to B, but now all these threads are stuck, because they are unable to make RPCs from B to A?</p>
</blockquote>
</aside>
<p>This is true for temporary <code>ProcessGroup</code> backend, but I am not 100% if this (send/recv share the same thread pool) is the case for <code>TensorPipe</code> backend. <a class="mention" href="/u/lcw">@lcw</a> could you please confirm?</p>
<p>The default RPC functions will block one thread on the callee side until the response is ready. If you would like the response being processed asynchronously, you can decorate the user function with <a href="https://pytorch.org/docs/master/rpc.html#torch.distributed.rpc.functions.async_execution" rel="noopener nofollow ugc"><code>@rpc.functions.async_execution</code></a> and let the user function return a <code>Future</code>. This should release the thread on callee as soon as it gets the <code>Future</code> object, and can resume running the callback on the <code>Future</code> when the <code>Future</code> is completed.</p>

lcw: <p>From what I can remember the TensorPipe agent does indeed only have one thread pool which is shared for many “purposes”, so what you said is totally reasonable.</p>
<p>I don’t immediately recognize the initial error message you got, as it mentions retries, but I don’t think we support retires in “standard” RPC messages. I think we only support them for RRef-related messages and other internal stuff. <a class="mention" href="/u/mrshenli">@mrshenli</a> Is that the case? <a class="mention" href="/u/jeremysalwen">@jeremysalwen</a> Are you using RRefs, dist autograd, or other such things? Unfortunately the message doesn’t say what the underlying failure is, but I suspect it could be a timeout?</p>
<p>Also note that as <a class="mention" href="/u/mrshenli">@mrshenli</a> said, it’s an antipattern to synchronously block in a remote RPC function or a callback. Doing so will block a thread and eventually lead to starvation. If you must do so, please ensure you’re doing it for a limited number of RPC calls, and size the thread pool accordingly. However, it would be better to use “native” asynchronous patterns.</p>

mrshenli: <aside class="quote no-group" data-username="lcw" data-post="5" data-topic="115550">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/b5e925/48.png" class="avatar"> lcw:</div>
<blockquote>
<p>I think we only support them for RRef-related messages and other internal stuff. <a class="mention" href="/u/mrshenli">@mrshenli</a> Is that the case?</p>
</blockquote>
</aside>
<p>Yep, this is true. We don’t yet have a good way to retry messages with user functions.</p>

jeremysalwen: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> I am using <span class="mention">@rpc.functions.async_execution</span> for the calls from B to A, but I don’t see an easy way to do so for the RPC from A to B.  Remember, internally the RPC from A to B then calls back to B multiple times, so I would need to suspend/resume the thread in order to free it.  (i.e. I would need something like asyncio, which would much further complicate things).</p>
<p><a class="mention" href="/u/lcw">@lcw</a> I am using RRefs, but not dist autograd, or any other advanced features.  I’m not sure how I would transform my code to be non-blocking.  Fundamentally, my RPC running on process B needs to wait until its RPC to process A completes before continuing execution.  I would expect that pytorch distributed should recognize this situation, and return the thread to the thread pool while it is waiting on the RPC.  What could I change in my code so that it “returns the thread” to the thread pool, but continues executing once the RPC completes?</p>

lcw: <blockquote>
<p>I would expect that pytorch distributed should recognize this situation, and return the thread to the thread pool while it is waiting on the RPC.</p>
</blockquote>
<p>Unfortunately that’s not something that can easily be achieved. It can be done through Python’s asyncio package but that’s not supported by PyTorch (for historical reasons I guess, and it’s hard now to retrofit it). Any other alternative would basically consist in reimplementing our own asyncio, and that’s just unreasonable.</p>
<blockquote>
<p>What could I change in my code so that it “returns the thread” to the thread pool, but continues executing once the RPC completes?</p>
</blockquote>
<p>I think your situation could be addressed by using the <code>async_execution</code> decorator, and futures, as <a class="mention" href="/u/mrshenli">@mrshenli</a> already mentioned. More concretely, here is what such a code could look like:</p>
<pre><code class="lang-auto"># Imagine this is the original code:
def my_function(a):
    b = foo(a)
    c = rpc.rpc_sync("other_worker", my_other_function, args=(b,))
    d = bar(c)
    return d

# It could become like this:
@rpc.functions.async_execution
def my_function(a):
    b = foo(a)
    fut_c = rpc.rpc_async("other_worker", my_other_function, args=(b,))
    fut_d = fut_c.then(lambda fut: bar(fut.value()))
    return fut_d
</code></pre>

jeremysalwen: <p>For reference, I ultimately decided to directly integrate asyncio with pytorch RPC to get around this issue.  I describe how I did it in this post: <a href="https://discuss.pytorch.org/t/pytorch-distributed-rpc-bottleneck-in-recursive-compile-class/117005/9" class="inline-onebox">Pytorch Distributed RPC bottleneck in _recursive_compile_class - #9 by jeremysalwen</a></p>
<p>This idea of using <code>@rpc.functions.async_execution</code> is interesting, but to me the example looks like it’s begging to be a coroutine instead of a pair of functions chained together with a callback <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>