Selecting action of N agents inside a single GPU with torch.distributed.rpc

deepfailure: <p>Hello,</p>
<p>I want to modify this code <a href="https://github.com/pytorch/examples/blob/master/distributed/rpc/rl/main.py" class="inline-onebox" rel="noopener nofollow ugc">examples/main.py at master · pytorch/examples · GitHub</a> which has 1 Agent and N observers which interact with an environment at the same time through torch.distributed.rpc.</p>
<p>My goal is to obtain N agents and 1 Simulator and the Simulator would ask the agents to sample actions and update when required.</p>
<p>For example to select action:</p>
<pre><code class="lang-auto">def select_actions_all_agents(self, state):

        self.current_actions = np.zeros(self.current_actions.shape, dtype=np.int32) * -1000
        futs = []
        start_time = time.time()
        for ag_rreff in self.ag_rrefs:
            # make async RPC to kick off an episode on all observers
            futs.append(
                rpc_async(
                    ag_rreff.owner(),
                    _call_method,
                    args=(Agent.select_action, ag_rreff, self.sim_rref, state)
                )
            )
            # wait until all agents have finished selecting action
            for fut in futs:
                fut.wait()

            self.time_select_action += (time.time() - start_time)
            self.num_time_select_action += 1
</code></pre>
<p>However, it seems that it does not reduce the inference time.</p>
<p>When instantiating each agent I send it inside the same GPU.</p>
<pre><code class="lang-auto">class Agent:
    def __init__(self):
        self.id = rpc.get_worker_info().id
        self.device = ("cuda" if torch.cuda.is_available() else "cpu")
        torch.manual_seed(args.seed+self.id)
        self.policy = Policy()
        self.policy.to(self.device)
</code></pre>
<p>I would expect that each agent would operate in parallel in the GPU and the inference time greatly reduced.</p>
<p>Any ideas?</p>

wayi: <p>Is this the same question as this post?</p><aside class="quote quote-modified" data-post="1" data-topic="121256">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/a3d4f5/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/rpc-does-not-seem-to-help-in-forward-time/121256">RPC does not seem to help in forward time</a> <a class="badge-category__wrapper " href="/c/distributed/distributed-rpc/20"><span data-category-id="20" data-parent-category-id="12" data-drop-close="true" class="badge-category --has-parent"><span class="badge-category__name">distributed-rpc</span></span></a>
  </div>
  <blockquote>
    Using RPC only decreases “backward time”. 
Hello I am using distributed.rpc changing the example in <a href="https://github.com/pytorch/examples/blob/01539f9eada34aef67ae7b3d674f30634c35f468/distributed/rpc/rl/main.py#L41" class="inline-onebox" rel="noopener nofollow ugc">examples/main.py at 01539f9eada34aef67ae7b3d674f30634c35f468 · pytorch/examples · GitHub</a> 
I have N agents and 1 simulation. I would like to select actions in parallel, if possible in different GPU’s. 
However, when I execute my code the only thing that seems to reduce is the update part. Below there is the code for selecting the actions which does not seem to be reduced at all, any ideas? 
from to…
  </blockquote>
</aside>