How to store embeddings from different ranks in DistributedDataParallel mode?

username: RocketFlash
cooked: <p>I want to run my model on dataset and store all embeddings using DistributedDataParallel. I created dataloader with DistributedSampler and now want to store all embeddings in the form:<br>
(image_name, embedding)</p>
<p>And after that I want to save them as csv or pickle file.</p>
<p>Will it be correct to create a global list and store data there or will there be conflicts with writing to the list?</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="RocketFlash" data-post="1" data-topic="94405">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rocketflash/48/28255_2.png" class="avatar"> RocketFlash:</div>
<blockquote>
<p>Will it be correct to create a global list and store data there or will there be conflicts with writing to the list?</p>
</blockquote>
</aside>
<p>By “global list”, you mean Python global variable? And this will create a global list per process? Who will be writing to the global list? BTW, any reason for not using <a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html#torch.nn.Embedding" rel="noopener nofollow ugc"><code>nn.Embedding</code></a>?</p>

username: RocketFlash
cooked: <p>Yes, by “global list” I mean global python variable. I am using mp.spawn to start distributed training, so I thought that the variables inside the executable file in this case are visible to all ranks. But after executing the code, nothing was written into the dict. What are the benefits of using nn.Embedding? I want to store image_name and embeddings.</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="RocketFlash" data-post="3" data-topic="94405">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rocketflash/48/28255_2.png" class="avatar"> RocketFlash:</div>
<blockquote>
<p>But after executing the code, nothing was written into the dict.</p>
</blockquote>
</aside>
<p>Right, global vars are per-process, so each spawned child process will have a different global var.</p>
<blockquote>
<p>What are the benefits of using nn.Embedding?</p>
</blockquote>
<p>One benefit is that you can then run lookup ops on GPU. And if you need to let the training process to update the embedding as well, using <code>nn.Embedding</code> will also make it easier.</p>
<blockquote>
<p>I want to store image_name and embeddings.</p>
</blockquote>
<p>If you would like to pass those data back to the main process, one option is to use the multiprocessing <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#reuse-buffers-passed-through-a-queue" rel="noopener nofollow ugc">SimpleQueue</a>. See the example below.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/cb26661fe4faf26386703180a9045e6ac6d157df/test/test_multiprocessing.py#L580-L600">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/cb26661fe4faf26386703180a9045e6ac6d157df/test/test_multiprocessing.py#L580-L600" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/cb26661fe4faf26386703180a9045e6ac6d157df/test/test_multiprocessing.py#L580-L600" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/cb26661fe4faf26386703180a9045e6ac6d157df/test/test_multiprocessing.py#L580-L600</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="580" style="counter-reset: li-counter 579 ;">
          <li>def test_event_multiprocess(self):</li>
          <li>    event = torch.cuda.Event(enable_timing=False, interprocess=True)</li>
          <li>    self.assertTrue(event.query())</li>
          <li></li>
          <li>    ctx = mp.get_context('spawn')</li>
          <li>    p2c = ctx.SimpleQueue()</li>
          <li>    c2p = ctx.SimpleQueue()</li>
          <li>    p = ctx.Process(</li>
          <li>        target=TestMultiprocessing._test_event_multiprocess_child,</li>
          <li>        args=(event, p2c, c2p))</li>
          <li>    p.start()</li>
          <li></li>
          <li>    c2p.get()  # wait for until child process is ready</li>
          <li>    torch.cuda._sleep(50000000)  # spin for about 50 ms</li>
          <li>    event.record()</li>
          <li>    p2c.put(0)  # notify child event is recorded</li>
          <li></li>
          <li>    self.assertFalse(event.query())</li>
          <li>    c2p.get()  # wait for synchronization in child</li>
          <li>    self.assertTrue(event.query())</li>
          <li>    p.join()</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I am trying to understand this requirement. In your application, is it like each subprocess will produce some image embedding independently and concurrently, and then you wanna save those?</p>

username: RocketFlash
cooked: <aside class="quote no-group" data-username="mrshenli" data-post="4" data-topic="94405">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>In your application, is it like each subprocess will produce some image embedding independently and concurrently, and then you wanna save those?</p>
</blockquote>
</aside>
<p>Yes, each subprocess generate embeddings from dataloader batches. I want to process all my data (generate embeddings) as fast as possible, this is why I want to use DistributedDataParallel. Process and after that save everything in one file.</p>