Using CUDA multiprocessing with single GPU

fvncc: <p>This page outlines that the multiprocessing module can be used with CUDA:</p>
<p><a href="http://pytorch.org/docs/master/notes/multiprocessing.html" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/docs/master/notes/multiprocessing.html</a></p>
<p>However CUDA generally has issues running multiple processes in paralell on one GPU:</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/31643570/running-more-than-one-cuda-applications-on-one-gpu" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1405658/cache" target="_blank" rel="nofollow noopener">
    <img alt="cache" src="https://www.gravatar.com/avatar/4fd7ae9d7f24f1a2e7870a6989cf5c08?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/31643570/running-more-than-one-cuda-applications-on-one-gpu" target="_blank" rel="nofollow noopener">Running more than one CUDA applications on one GPU</a>
</h4>

<div class="tags">
  <strong>cuda, gpu, gpgpu, nvidia</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/1405658/cache" target="_blank" rel="nofollow noopener">
    cache
  </a>
  on <a href="https://stackoverflow.com/questions/31643570/running-more-than-one-cuda-applications-on-one-gpu" target="_blank" rel="nofollow noopener">12:55AM - 27 Jul 15</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Do these limitations apply to the pytorch multiprocessing module?</p>
<p>Thanks in advance <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

smth: <p>See <a href="http://pytorch.org/docs/master/notes/multiprocessing.html#sharing-cuda-tensors">http://pytorch.org/docs/master/notes/multiprocessing.html#sharing-cuda-tensors</a></p>
<p>You can use CUDA + multiprocessing if you use python3 and a different start method.</p>

fvncc: <p>Thanks, I see how to use CUDA with multiprocessing. However I would guess the most common use case of CUDA multiprocessing is utilizing multiple GPU’s (i.e. with one process on each GPU). If you want to train multiple small models in parallel on a single GPU, is there likely to be significant performance improvement over training them sequentially?</p>

smth: <p>GPUs dont do very well with multiple workloads / multiple threading models. So it’s almost always better to use one GPU per model. Unless your model is REALLY REALLY tiny… (to a point where CPU probably is faster)</p>

aman_bharat: <p>Hi,<br>
I am trying to run multiprocessing in my python program. I created two processes and passed a neural network in the one process and some heavy computational function in the other. I wanted the neural net to run on GPU and the other function on CPU and thereby I defined neural net using cuda() method. But when I run the program the got the following error:<br>
<strong>RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method</strong></p>
<p>So I tried with <strong>spawn</strong> as well as <strong>forkserver</strong> start method, but then I got the other error:<br>
<strong>RuntimeError: cuda runtime error (71) : operation not supported at …/torch/csrc/generic/StorageSharing.cpp:245</strong></p>
<p>I have tried python3 multiprocessing and torch.multiprocessing both but nothing worked for me.</p>

Syun: <p>Hi <a class="mention" href="/u/aman_bharat">@aman_bharat</a>,<br>
I have a similar problem with you.<br>
Did you solved the problem?<br>
If you could give me the solution, it would be very appreciated.<br>
Thanks in advance.</p>