PyTorch RPC multiple threading training

LW-Ricarido: <p>Hello!<br>
I try to build a distributed RL with PyTorch RPC, but I got a problem that I can’t use my GPU power fully. Here is my code structure. They are two parts, learner and actor.</p>
<pre><code class="lang-auto">class Learner:
    
    def __init__(self):
        self.policy = Policy()
        self.buffer = container()
        run_multiple_actor()
    
    def learning_loop(self):
        "backward &amp; optimize"
        while True:
            get_trans_from_buffer()
            computing_loss()
            backward()
            optimizer.step()
    
    def get_action(self, state):
        action = self.policy(state)
        insert_to_buffer(state,action)
        return action

class Actor:
    
    def __init__(self):
        self.env = env
        self.learner_rref = rref
    
    def act_loop(self):
        while True:
            state = self.env.step(action)
            action = self.learner_rref.run_sync().get_action(state)
</code></pre>
<p>For learner, after initating, it runs in <code>learning_loop</code>. For actor, after initating, it runs in <code>act_loop</code> and call Learner’s <code>get_action</code> from remote.</p>
<p>The question is that could <code>get_action</code> threads run simultaneously on GPU? If they can, tt seems that as long as I run enough actors, I fully use my GPU power. However, after adding serveal actors, the volatile of GPU stop inscreasing and stays in a low level(eg. 20% or 30%). And I don’t think it’s a problem of my CPU cores. I have enough CPU to make all actors run simultaneously.</p>
<p>Could anyone point out what’s the problem of my code. I am new to PyTorch PRC, help me please.</p>

mrshenli: <p>Hey <a class="mention" href="/u/lw-ricarido">@LW-Ricarido</a>, sorry about the delay.</p>
<p>If you use regular Python functions as RPC target, multiple requests cannot run in parallel on the callee side due to Python Global Interpreter Lock (GIL). To avoid the lock, you can convert your function into a TorchScript function by adding a <code>@torch.jit.script</code> decorator. Some examples are available <a href="https://pytorch.org/docs/master/rpc.html" rel="noopener nofollow ugc">here</a>, please search for <code>@torch.jit.script</code>.</p>
<p><a href="https://pytorch.org/docs/stable/jit.html" rel="noopener nofollow ugc">This</a> is a more thorough doc for TorchScript.<br>
Example tests can be found <a href="https://github.com/pytorch/pytorch/blob/master/torch/testing/_internal/distributed/rpc/jit/rpc_test.py" rel="noopener nofollow ugc">here</a>.</p>

LW-Ricarido: <p>Hey <a class="mention" href="/u/mrshenli">@mrshenli</a>, thanks a lot.  But I find some problems when I use <code>@torch.jit.script</code> decorator. When I use <code>torch.jit.script</code> , the complier cannot infer <code>self.policy</code> correctly, which is a <code>nn.Module</code> and an attribute of <code>Learner</code>. Should change class <code>Learner</code> and <code>Actor</code> to <code>torch.jit.ScriptModule</code>. And if I want to avoid GIL, should both callee function and caller be wrapped by <code>@torch.jit.script</code>?</p>

mrshenli: <p>Hey <a class="mention" href="/u/lw-ricarido">@LW-Ricarido</a></p>
<blockquote>
<p>And if I want to avoid GIL, should both callee function and caller be wrapped by  <code>@torch.jit.script</code> ?</p>
</blockquote>
<p>Yep, current RPC system requires caller and callee functions to be the same. You can, for example, define those functions in a utility file and let both caller and callee import that file.</p>