Pytorch MP manager dict hang in linux

Lewis_Liu: <p>I’m trying to share the weights of networks between processes using multiprocessing manager.dict<br>
The code is as follows</p>
<pre><code>    for name in critic_state_dict:
        self.shared_networks.critic[name] = T.tensor(critic_state_dict[name].clone().cpu().detach().numpy())
</code></pre>
<p>This works fine in windows. But when I use a cluster, it hangs in the middle of the for loop</p>
<p>How do I fix this? Or if I want to periodically share the weights among processes, how to do it properly?<br>
Thanks</p>

mrshenli: <p>Hey <a class="mention" href="/u/lewis_liu">@Lewis_Liu</a>,</p>
<p>Did you use fork or spawn?</p>
<blockquote>
<p>Or if I want to periodically share the weights among processes, how to do it properly?</p>
</blockquote>
<p>One solution is to create a multiprocessing <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#reuse-buffers-passed-through-a-queue" rel="noopener nofollow ugc">queue</a>, and pass that queue to child processes. Then, in the loop, use that queue to pass shared tensors. The test below can serve as an example:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/fc8bca094cc23a2394214c5cdbc8392a3d279e8c/test/distributed/test_c10d_spawn.py#L166-L184">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/fc8bca094cc23a2394214c5cdbc8392a3d279e8c/test/distributed/test_c10d_spawn.py#L166-L184" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/fc8bca094cc23a2394214c5cdbc8392a3d279e8c/test/distributed/test_c10d_spawn.py#L166-L184" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/fc8bca094cc23a2394214c5cdbc8392a3d279e8c/test/distributed/test_c10d_spawn.py#L166-L184</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="166" style="counter-reset: li-counter 165 ;">
          <li>@classmethod</li>
          <li>def _test_allgather_process(</li>
          <li>        cls, rank, filename, shared_tensors, world_size, init_pg, c2p, p2c):</li>
          <li>    pg = init_pg(rank, filename, world_size)</li>
          <li>    xs = [shared_tensors[rank]]</li>
          <li>    ys = [[torch.zeros_like(xs[0]) for i in range(world_size)]]</li>
          <li>    pg.allgather(ys, xs).wait()</li>
          <li>    for i in range(world_size):</li>
          <li>        c2p.put((rank, torch.ones(2, 2) * i, ys[0][i].to("cpu")))</li>
          <li></li>
          <li>    p2c.get()</li>
          <li></li>
          <li>@unittest.skipIf(not TEST_MULTIGPU, "At least 2 CUDA GPUS needed")</li>
          <li>def test_shared_allgather_gloo(self):</li>
          <li>    self._test_multiprocess(</li>
          <li>        ProcessGroupShareTensorTest._test_allgather_process,</li>
          <li>        [torch.ones(2, 2).to(i) * i for i in range(self.world_size)],</li>
          <li>        ProcessGroupShareTensorTest._init_pg_gloo,</li>
          <li>        self.world_size)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Lewis_Liu: <p>Hi Li,</p>
<p>I switched to using Queue. But I cannot avoid firstly getting the net tensors from state_dict right?</p>
<p>I believe it’s spawn on my windows workstation and it’s fork on the linux cluster if i’m correct</p>

mrshenli: <aside class="quote no-group" data-username="Lewis_Liu" data-post="3" data-topic="89644">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lewis_liu/48/12953_2.png" class="avatar"> Lewis_Liu:</div>
<blockquote>
<p>I switched to using Queue. But I cannot avoid firstly getting the net tensors from state_dict right?</p>
</blockquote>
</aside>
<p>I don’t have the full context here. Can you let the processing holding the state_dict be the writer to the queue?</p>

Lewis_Liu: <p>Yep.</p>
<p>The network is trained and updated for a step. After this, the process has only one sole task that is to write the state_dict into the queue. Other processes doesn’t have direct access to the network except through the queue</p>