Access data of a GPU process from main

username: TT_YY
cooked: <p>Hi.</p>
<p>I have a program using distributed data parallel of Pytorch.<br>
It’s working well, but I do not know how to access data in the GPU process from main().</p>
<p>In particular, a GPU process, train(), produces a list of loss, and I want to plot it in the main() after returning from spawn(). However, I do not know how to access the list in train() on GPU from main() on CPU.</p>
<p>If I use a global variable, it should work, but it does not seem to be the best answer. I understand that printing loss can be done by gpu[0], and maybe even plotting the graph too. But, I want to do many tasks to analyze the results in main().</p>
<p>I appreciate any information or examples. Thank you.</p>

username: TT_YY
cooked: <p>I have checked communication using args and global variables.<br>
Non of them works for transmitting data from a  GPU process to main(), when I use DDP.</p>
<p>Usually, the arguments hold pointers to variables, and main() and called functions can share the same variables. But, when the processes are spawned in DDP, it seems that args are deep-copied and there is no common variables having the same ids between main() and the processes.</p>
<p>A global variable can be declared in the process, but it’s not shared with main().</p>
<p>Does everybody using DDP plot the loss charts in a spawned process on GPU?<br>
I have no idea how to send the loss list from the process to main().</p>
<p>Please advise.</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="TT_YY" data-post="2" data-topic="95529">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tt_yy/48/25673_2.png" class="avatar"> TT_YY:</div>
<blockquote>
<p>I have checked communication using args and global variables.<br>
Non of them works for transmitting data from a GPU process to main(), when I use DDP.</p>
</blockquote>
</aside>
<p>This is true, because Python global vars are per-process concept.</p>
<blockquote>
<p>Does everybody using DDP plot the loss charts in a spawned process on GPU?</p>
</blockquote>
<p>This can be done using <code>torch.multiprocessing.SimpleQueue</code>. E.g., let the main process create the queue, pass it to the child process, and then let the child process put the loss object to the queue. Then, the main process should be able to see that.</p>
<p>The test below can serve as an example:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/2c4b4aa81bc8dba8272e9c7190edcaa3e114ec15/test/test_multiprocessing.py#L580-L600">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/2c4b4aa81bc8dba8272e9c7190edcaa3e114ec15/test/test_multiprocessing.py#L580-L600" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/2c4b4aa81bc8dba8272e9c7190edcaa3e114ec15/test/test_multiprocessing.py#L580-L600" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/2c4b4aa81bc8dba8272e9c7190edcaa3e114ec15/test/test_multiprocessing.py#L580-L600</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="580" style="counter-reset: li-counter 579 ;">
          <li>def test_event_multiprocess(self):</li>
          <li>    event = torch.cuda.Event(enable_timing=False, interprocess=True)</li>
          <li>    self.assertTrue(event.query())</li>
          <li></li>
          <li>    ctx = mp.get_context('spawn')</li>
          <li>    p2c = ctx.SimpleQueue()</li>
          <li>    c2p = ctx.SimpleQueue()</li>
          <li>    p = ctx.Process(</li>
          <li>        target=TestMultiprocessing._test_event_multiprocess_child,</li>
          <li>        args=(event, p2c, c2p))</li>
          <li>    p.start()</li>
          <li></li>
          <li>    c2p.get()  # wait for until child process is ready</li>
          <li>    torch.cuda._sleep(50000000)  # spin for about 50 ms</li>
          <li>    event.record()</li>
          <li>    p2c.put(0)  # notify child event is recorded</li>
          <li></li>
          <li>    self.assertFalse(event.query())</li>
          <li>    c2p.get()  # wait for synchronization in child</li>
          <li>    self.assertTrue(event.query())</li>
          <li>    p.join()</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: seungjun
cooked: <p>I’m not sure if my case is what you want but I use GPU communication functions to plot loss graph during training.</p>
<p>I use a custom Loss class that inherits nn.modules.loss._Loss.<br>
It calculates the loss and stores the record and plots the loss graph.<br>
The loss values are synchronized inside the Loss class<br>
Thus, I don’t have to send values to main() scope.</p>
<p>Here’s my public <a href="https://github.com/SeungjunNah/DeepDeblur-PyTorch/blob/master/src/loss/__init__.py#L259" rel="nofollow noopener">github code</a>.</p>
<p>all_reduce function performs sync and it is called at the end of an epoch from trainer.</p>

username: TT_YY
cooked: <p>Thank you, Shen Li.</p>
<p>This is be what I was looking for. I will try the sample code and try to use SimpleQueue() in my program. I hope it works!</p>
<p>Thank you.</p>

username: TT_YY
cooked: <p>Thank you, <a href="https://discuss.pytorch.org/u/seungjun">seungjun</a></p>
<p>I appreciate your code.<br>
I understand that all_reduce function transmits loss between GPUs.</p>
<p>I am trying to evaluate performance of optimizers by changing many factors such as batch size and learning rate. Therefore, I have multiple loops to change the variables outside the optimization loop. Also, I have to collect many kinds of data along with loss and accuracy to analyze the details of the optimizers and plot them.</p>
<p>I felt a kind of odd about performing such non-multiplication looping tasks and all plotting using expensive GPU and its memory. So, I tried to implement all the outer loops, analysis, and plotting tasks in main(), which requires getting the information from the spawned GPU processes.</p>
<p>Thank you.</p>