Can RPC leverage multicore?

username: kmaeng
cooked: <p>I am using torch.distributed.rpc. I can set the rpc to have many threads using <strong>rpc_backend_options</strong>, but it seems like it is not being mapped onto idle CPUs that I have.<br>
Specifically, to test out, I’ve sent 1-4 asynchronous RPC calls to a server which has 80 CPUs.<br>
Below is the code for reference.</p>
<pre><code class="lang-auto">import os
import time
import torch
import torch.nn as nn
import numpy as np
from torch.multiprocessing import Process
import torch.distributed.rpc as rpc

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.l0 = nn.Linear(2, 2)
        W = np.random.normal(0, 1, size=(2,2)).astype(np.float32)
        self.l0.weight.data = torch.tensor(W, requires_grad=True)
        self.l1 = nn.Linear(2, 2)
        W = np.random.normal(0, 1, size=(2,2)).astype(np.float32)
        self.l1.weight.data = torch.tensor(W, requires_grad=True)

    def forward(self, x):
        return self.l1(self.l0(x))

def test(t):
    print("RPC called")
    for i in range(100000):
        t2 = t*1.000001
    return t2

def run(i):
    rpc.init_rpc("Rank"+str(i), rank=i, world_size=2)
    if i == 0:
        with torch.autograd.profiler.profile(True, False) as prof:
            net = Net()
            optimizer = torch.optim.SGD(net.parameters(), lr=0.1)
            input = torch.tensor([1.0, 2.0])
            reqs = []
            reqs.append(rpc.rpc_async("Rank1", test, args=(input,)))
            reqs.append(rpc.rpc_async("Rank1", test, args=(input,)))
            reqs.append(rpc.rpc_async("Rank1", test, args=(input,)))
            reqs.append(rpc.rpc_async("Rank1", test, args=(input,)))
            #reqs.append(rpc.rpc_async("Rank1", test, args=(input,)))
            for req in reqs:
                input += req.wait()
            print("RPC Done")
            y = net(input)
            optimizer.zero_grad()
            y.sum().backward()
            optimizer.step()
        print(prof.key_averages().table(sort_by="cpu_time_total"))
        prof.export_chrome_trace("test.json")
    else:
        pass

    rpc.shutdown()

if __name__ == "__main__":
    os.environ['MASTER_ADDR'] = "localhost"
    os.environ['MASTER_PORT'] = "29500"
    ps = []
    for i in [0, 1]:
        p = Process(target=run, args=(i,))
        p.start()
        ps.append(p)

    for p in ps:
        p.join()
</code></pre>
<p>As you can see, I am just doing some compute-intensive work on the server using RPC.<br>
Below is the result from my profiler.<br>
When I do 1 RPC call:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/2/0225b05c5f78dc7da0ac4d682f45ca63509e104e.png" data-download-href="https://discuss.pytorch.org/uploads/default/0225b05c5f78dc7da0ac4d682f45ca63509e104e" title="Screen Shot 2020-05-27 at 8.58.57 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/0225b05c5f78dc7da0ac4d682f45ca63509e104e_2_690x99.png" alt="Screen Shot 2020-05-27 at 8.58.57 PM" data-base62-sha1="iZHAybQnkdWkXxzCOfWEVwnMia" width="690" height="99" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/0225b05c5f78dc7da0ac4d682f45ca63509e104e_2_690x99.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/0225b05c5f78dc7da0ac4d682f45ca63509e104e_2_1035x148.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/0225b05c5f78dc7da0ac4d682f45ca63509e104e_2_1380x198.png 2x" data-dominant-color="EDEFEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-05-27 at 8.58.57 PM</span><span class="informations">3226×464 42.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
When I do 4 RPC calls:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b04d85d3d1bae1f964d8964c0c120bd3a2d0d35b.png" data-download-href="https://discuss.pytorch.org/uploads/default/b04d85d3d1bae1f964d8964c0c120bd3a2d0d35b" title="Screen Shot 2020-05-27 at 8.59.42 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b04d85d3d1bae1f964d8964c0c120bd3a2d0d35b_2_690x70.png" alt="Screen Shot 2020-05-27 at 8.59.42 PM" data-base62-sha1="p9E7QtfgjNmxyTJgraePMRwQh7B" width="690" height="70" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b04d85d3d1bae1f964d8964c0c120bd3a2d0d35b_2_690x70.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b04d85d3d1bae1f964d8964c0c120bd3a2d0d35b_2_1035x105.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b04d85d3d1bae1f964d8964c0c120bd3a2d0d35b_2_1380x140.png 2x" data-dominant-color="D4E1C8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-05-27 at 8.59.42 PM</span><span class="informations">3208×326 49.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Default RPC init makes 4 send_recv_threads. So it should be able to “concurrently” run my 4 RPC requests. However, as you can see, the time to finish the RPC requests grew almost linearly (from 460ms to 2200ms) with 4 requests, meaning that they are using only one core and are not being processed in parallel (i.e., concurrent, but not parallel).</p>
<p>I know that python threads (unlike processes) cannot execute in parallel on different cores. Is RPC threads also (because they are threads) cannot run in parallel on different cores?<br>
Is there a way to run different RPC request received on different cores? Or should I manually spawn processes in the receiving server side to run the requests in parallel and leverage my multicore server?</p>
<p>Thank you.</p>

username: pritamdamania87
cooked: <p>This is likely due to Python GIL on the server side. Can you torchscript (<a href="https://pytorch.org/docs/stable/jit.html" rel="nofollow noopener">https://pytorch.org/docs/stable/jit.html</a>) the <code>test</code> method and try again? That should avoid GIL.</p>

username: iffiX
cooked: <p>Probably <strong>Yes</strong> for torch jit scripts, and <strong>No</strong> for regular python functions.</p>
<p>I will describe my understanding of the framework in details. For c++ files, their root directory is (git master) <code>torch/csrc/distributed</code>.</p>
<p>Internally, torch rpc framework does the following things:</p>
<ol>
<li>
<p>entry point: <code>rpc.async</code>, <code>rpc.sync</code> use the same <code>_invoke_rpc()</code> while <code>rpc.remote</code> use its own implementation. For these three methods, they all categorize your calls into three categories: <code>builtin</code> is for python builtin function, <code>udf</code> is for user defined functions, <code>jit</code> is for torch script. Then all three lower level calls go into the torch C library.</p>
</li>
<li>
<p>c library: the c-python interface is defined in <code>rpc/init.cpp</code>, which uses <code>pybind11</code> to define the interfaces, on calling the interface, call guards (constructed before wrapped functions) are <code>gil_scoped_release</code> so GIL is released here for all three interface categories. The wrapped functions are defined in <code>rpc/python_functions.cpp</code> they will find your target rpc process (c++ class <code>RpcAgent</code>) and send your message to it. GIL will be reaquired after finishing the c function call due to the automatic deconstruction of call gaurds.</p>
</li>
<li>
<p>rpc agent: <code>RpcAgent</code> class will initialize <code>cb_</code> member on construction, which is a <code>unique_ptr</code> of type <code>RequestCallback</code>, there are two derived classes of <code>RpcAgent</code>: <code>TensorPipeAgent</code> and <code>ProcessGroupAgent</code>. In the rpc use case you are dealing with <code>ProcessGroupAgent</code>, in its member function <code>handleRecv()</code> it will use the <code>cb_</code>member to handle the call back. All these agents are defined in <code>rpc/&lt;agent_name_lower_case&gt;.cpp</code> so you should be able to find them easily.</p>
</li>
<li>
<p>request callback: Request callback is an abstract functor defined in <code>request_callback.h</code> and <code>request_callback.cpp</code>, it has a virtual <code>processMessage()</code> method. Its real implementation in <code>request_callback_impl.cpp</code> defines this method and calls its member method <code>processRpc()</code>.</p>
</li>
<li>
<p>process rpc: This function handles rpc calls based on its message type, <code>SCRIPT_CALL</code> and <code>SCRIPT_REMOTE_CALL</code> will handle jit scripted calls, however in <code>PYTHON_CALL</code>:</p>
<pre><code>  {
       py::gil_scoped_acquire acquire;
       serializedPyObj =
       std::make_shared&lt;SerializedPyObj&gt;(pythonRpcHandler.serialize(
       pythonRpcHandler.runPythonUdf(std::move(upc).movePythonUdf())));
  }
</code></pre>
</li>
</ol>
<p>There is a <code>py::gil_scoped_acquire</code>, according to the pybind11 definition, this will hold the gil lock until <code>acquire</code> is deconstructed (aka leaving this c++ scope), so <strong>Nope</strong>, you cannot leverage multicore by using multi-threads used in rpc.</p>
<p><strong>Note</strong> this explanation is valid for commit hash <code>176174a68ba2d36b9a5aaef0943421682ecc66d4</code> and release till 1.5.0, as I can see in their source code that they are planning to further abstract away the switch case in <code>processRpc()</code> to an abstract <code>execute()</code> method of <code>RpcCommandBase</code></p>

username: iffiX
cooked: <p>So your code is theoretically equivalent to using the ThreadPool(4).map from multiprocessing library. The test code is as follows:</p>
<pre><code class="lang-auto">import torch as t
from multiprocessing.pool import ThreadPool
from utils.helper_classes import Timer

def test(t):
    print("RPC called")
    for i in range(100000):
        t2 = t*1.000001
    return t2

def test1():
    tm = Timer()
    ts=t.Tensor([1,2])
    tm.begin()
    for i in range(4):
        test(ts)
    print(tm.end())

def test2():
    tm = Timer()
    ts = t.Tensor([1,2])
    pool = ThreadPool(4)
    tm.begin()
    pool.map(test, (ts, ts, ts, ts))
    print(tm.end())
</code></pre>
<p>And my test result is:<br>
test1: 2.091 s<br>
test2: 2.404 s</p>
<p>You can dispatch sub processes in the rpc call to work around the GIL.</p>