One GPU inference with one model and merge four times results together

username: kehuantiantang
cooked: <p>I have four gpus, I make one program to load four models and inference each model with shared data in each single gpus, then merge each model’s results. For example:</p>
<pre><code class="lang-python">models = [model1.to('cuda:0'), model2.to('cuda:1'), model3.to('cuda:2')]
data = get_data()
results = []
for index, model in enumerate(models):
    data = data.to('cuda:%d'%index)
    r = model(data)
    result.append(r)
final_result = fusion_result(result)
</code></pre>
<p>But right now, each model can only get data from a single process, can I get data and share to multi-model and do <strong>asynchronous inference</strong> by multiprocessing? Many thanks.</p>

username: ptrblck
cooked: <p>Since the models and data tensors are already pushed to different devices, each execution is already performed asynchronously, which should also be visible in e.g. <code>nvprof</code> or <code>NSIGHT</code>.</p>

username: kehuantiantang
cooked: <p>According to my code, the GPU will do inference only call <code>model(data)</code> ， which means if my model is so complex, another three gpu will stop running for a while until one gpu finish inference. I am wonder that can those inferences doing at the same time with shared data? Thank you so much.</p>

username: ptrblck
cooked: <p><code>model(data)</code> will execute the forward pass on the used device inside the model.<br>
If your model implementation is only using a single device (i.e. no model sharding is used), the execution will begin asynchronously.<br>
The next <code>model(data)</code> call, will launch the execution on the next specified device (assuming you’ve created copies of the model and data on other devices).</p>

username: kehuantiantang
cooked: <p>Yes, but the remaining device will <strong>suspend until the next</strong> <code>model(data)</code> call, which spends a lot of time. What I want to do right now is send the shared data to different devices at the same time, then each device will execute the <code>model(data)</code> forward asynchronously and merge the result in the host process. Any idea to do that?  Thank you so much for your reply.</p>

username: mrshenli
cooked: <p>You can use <code>SimpleQueue</code> in <code>torch.multiprocessing</code> to do that. E.g., you can create a queue between the host process and each subprocess, and use the queue to pass input data and collect output. The test below can serve as an example:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/0c5cd8c2b9cdf473e30bbb1b49ca80ed442813df/test/test_multiprocessing.py#L577-L600" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/0c5cd8c2b9cdf473e30bbb1b49ca80ed442813df/test/test_multiprocessing.py#L577-L600" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/0c5cd8c2b9cdf473e30bbb1b49ca80ed442813df/test/test_multiprocessing.py#L577-L600</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="577" style="counter-reset: li-counter 576 ;">
<li>@unittest.skipIf(NO_MULTIPROCESSING_SPAWN, "Disabled for environments that \</li>
<li>                 don't support multiprocessing with spawn start method")</li>
<li>@unittest.skipIf(not TEST_CUDA_IPC, 'CUDA IPC not available')</li>
<li>def test_event_multiprocess(self):</li>
<li>    event = torch.cuda.Event(enable_timing=False, interprocess=True)</li>
<li>    self.assertTrue(event.query())</li>
<li>
</li><li>    ctx = mp.get_context('spawn')</li>
<li>    p2c = ctx.SimpleQueue()</li>
<li>    c2p = ctx.SimpleQueue()</li>
<li>    p = ctx.Process(</li>
<li>        target=TestMultiprocessing._test_event_multiprocess_child,</li>
<li>        args=(event, p2c, c2p))</li>
<li>    p.start()</li>
<li>
</li><li>    c2p.get()  # wait for until child process is ready</li>
<li>    torch.cuda._sleep(50000000)  # spin for about 50 ms</li>
<li>    event.record()</li>
<li>    p2c.put(0)  # notify child event is recorded</li>
<li>
</li></ol></code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/0c5cd8c2b9cdf473e30bbb1b49ca80ed442813df/test/test_multiprocessing.py#L577-L600" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: kehuantiantang
cooked: <p>Thank you so much, I find that the pickle and unpickle spend a lot of time when <strong>transfer the numpy image to subprocessing <code>queue.put(img)</code> and get result <code>queue.get() </code></strong>. It is not efficient than loop (in single process). Do you have some idea to save this problem ? Many thanks.</p>

username: mrshenli
cooked: <p>Does using a <code>shared_memory</code> tensor help in this case? See the doc below:</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" target="_blank" rel="noopener nofollow ugc">Multiprocessing best practices — PyTorch 1.6.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: VitalyFedyunin
cooked: <p><a class="mention" href="/u/kehuantiantang">@kehuantiantang</a> can you please clarify the nature of the <code>data</code>, is it pinned memory on CPU or it is on GPU? It might be more efficient to put it on one of the GPU devices and share it with others via IPC instead of transferring it from device to device using <code>.to()</code> call.</p>
<p>Also it seems that you are looking for solution to evenly distribute load between GPUs as one of the models are much faster than others. In this case I recommend to try outer loop which will rotate models  between GPUs (if memory allows) and avoid <code>fusion_result</code> calls as long as possible as it is looks like your synchronization point.</p>

username: kehuantiantang
cooked: <p>Sorry. I am not clarify so clearly, my data is a numpy image which shape is 416x416x3, because my model is so large, it cannot load into one gpu at same time, so I use <code>model1.to(cuda:0)</code>, <code>data.to('cuda:0)</code> and rotate models between GPUs by <code>for loop</code>. seem like that:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a08815eadf7c5a5d125ea36ef9848ab387806a51.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a08815eadf7c5a5d125ea36ef9848ab387806a51" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a08815eadf7c5a5d125ea36ef9848ab387806a51_2_687x500.jpeg" alt="image" data-base62-sha1="mU7Xm57pj7DEIDqJMRNPLP5vuQ9" width="687" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a08815eadf7c5a5d125ea36ef9848ab387806a51_2_687x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/a/0/a08815eadf7c5a5d125ea36ef9848ab387806a51.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/0/a08815eadf7c5a5d125ea36ef9848ab387806a51.jpeg 2x" data-dominant-color="1F273F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">891×648 234 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
All the model run in single processes, and can only inference in one GPU(other will suspend).</p>

username: kehuantiantang
cooked: <p>Yes, I try this one,  to deliver the data to each sub-process by queue.  I find that queue need to <code>pickle</code> and <code>unpickle</code> operation, but my data is numpy image which has shape 416x416x3. The <code>pickle</code> and <code>unpickle</code> speed a lot of time. It is not efficient than <code>for loop</code>, so I give up. Thanks a lot.</p>