Return from mp.spawn()

arinaruck: <p>Hi!<br>
I am using a <code>nn.parallel.DistributedDataParallel</code> model for both training and inference on multiple gpu.<br>
To achieve that I use<br>
<code>mp.spawn(evaluate, nprocs=n_gpu, args=(args, eval_dataset))</code><br>
To evaluate I actually need to first run the dev dataset examples through a model and then to aggregate the results. Therefore I need to be able to return my predictions to the main process (possibly in a dict, but some other data structure should work as well). I’ve tried providing an extra dict argument <code>mp.spawn(evaluate, nprocs=n_gpu, args=(args, eval_dataset, out_dict))</code> and modifying it in the function but apparently spawn copies it, so the dict in the main process is not modified.<br>
I guess, I could write the results to the file and then read in the main process but it doesn’t seem like the most elegant solution. Is there a better way to return values from spawned functions?<br>
Thanks!</p>

mrshenli: <blockquote>
<p>Is there a better way to return values from spawned functions?</p>
</blockquote>
<p>If you want to pass the result from spawned processes back to the parent process, you can let the parent process create <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#reuse-buffers-passed-through-a-queue" rel="noopener nofollow ugc">multiprocessing queues</a>, pass it to children processes, and let children processes send result back through the queue. See the following code:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/cb26661fe4faf26386703180a9045e6ac6d157df/test/test_multiprocessing.py#L577-L600">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/cb26661fe4faf26386703180a9045e6ac6d157df/test/test_multiprocessing.py#L577-L600" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/cb26661fe4faf26386703180a9045e6ac6d157df/test/test_multiprocessing.py#L577-L600" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/cb26661fe4faf26386703180a9045e6ac6d157df/test/test_multiprocessing.py#L577-L600</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="577" style="counter-reset: li-counter 576 ;">
          <li>@unittest.skipIf(NO_MULTIPROCESSING_SPAWN, "Disabled for environments that \</li>
          <li>                 don't support multiprocessing with spawn start method")</li>
          <li>@unittest.skipIf(not TEST_CUDA_IPC, 'CUDA IPC not available')</li>
          <li>def test_event_multiprocess(self):</li>
          <li>    event = torch.cuda.Event(enable_timing=False, interprocess=True)</li>
          <li>    self.assertTrue(event.query())</li>
          <li></li>
          <li>    ctx = mp.get_context('spawn')</li>
          <li>    p2c = ctx.SimpleQueue()</li>
          <li>    c2p = ctx.SimpleQueue()</li>
          <li>    p = ctx.Process(</li>
          <li>        target=TestMultiprocessing._test_event_multiprocess_child,</li>
          <li>        args=(event, p2c, c2p))</li>
          <li>    p.start()</li>
          <li></li>
          <li>    c2p.get()  # wait for until child process is ready</li>
          <li>    torch.cuda._sleep(50000000)  # spin for about 50 ms</li>
          <li>    event.record()</li>
          <li>    p2c.put(0)  # notify child event is recorded</li>
          <li></li>
      </ol>
    </code></pre>


  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/cb26661fe4faf26386703180a9045e6ac6d157df/test/test_multiprocessing.py#L577-L600" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>If the result does not have to go back to the parent process, you can use <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.gather" rel="noopener nofollow ugc">gather</a> or <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_gather" rel="noopener nofollow ugc">allgather</a> to communicate the result across children processes.</p>

xannex: <p>Hi Arina. What was your solution at the end?</p>

arinaruck: <p>Hi, for me writing to files actually worked out quite alright.<br>
I’m just using <code>pickle.dump</code> in the spawned processes and <code>pickle.load</code> in the main one</p>

arinaruck: <p>Actually met this problem once again recently and decided to make use of the mp.Queue as <a class="mention" href="/u/mrshenli">@mrshenli</a> suggested, so I used</p>
<pre><code class="lang-auto">result_queue = mp.Queue()
for rank in range(args.n_gpu):
        mp.Process(target=get_features, args=(rank, args, eval_dataset, result_queue)).start()
</code></pre>
<p>to start the processes instead of mp.spawn and just added the results in my queue with</p>
<pre><code class="lang-auto">result_queue.put((preds, labels, ranks))
</code></pre>
<p>in those processes.<br>
To later collect the results I did the following:</p>
<pre><code class="lang-auto">        for _ in range(args.n_gpu):
            temp_result = result_queue.get()
            preds.append(temp_result[0])
            labels.append(temp_result[1])
            ranks.append(temp_result[2])
            del temp_result
</code></pre>