Multiprocessing for cuda error

username: KelleyYin
cooked: <p>I want to use torch.multiprocessing to accelerate my loop, however there are some errors .<br>
I can’t absolutely understand the shared cuda menmery for subprocess .<br>
Does anyone give some explanations ?</p>
<pre><code class="lang-auto">from torch.multiprocessing import Pool

def use_gpu():
    t = []
    for i in range(5):
        time.sleep(1)
        a = torch.randn(1000, 1000).cuda(3)      
        t.append(a)
    return t

if __name__ == "__main__":
    # torch.cuda.set_device(3)
    pool = Pool()
    result = []
    a = time.time()
    for i in range(10):
        result.append(pool.apply_async(use_gpu))
    pool.close()
    pool.join()
    print("cost time :", time.time() - a)
</code></pre>
<p>This snippet can work well .<br>
if I used <code>torch.cuda.set_device(3)</code> for allocation gpu, there are some errors as follow .</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCGeneral.c line=74 error=3 : initialization error
THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCGeneral.c line=74 error=3 : initialization error
THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCGeneral.c line=74 error=3 : initialization error
THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCGeneral.c line=74 error=3 : initialization error
THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCGeneral.c line=74 error=3 : initialization error
THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCGeneral.c line=74 error=3 : initialization error
THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCGeneral.c line=74 error=3 : initialization error
THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCGeneral.c line=74 error=3 : initialization error
THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCGeneral.c line=74 error=3 : initialization error
THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCGeneral.c line=74 error=3 : initialization error

</code></pre>
<p>What cause it and how to solve it ?<br>
Thanks in advance .</p>

username: smth
cooked: <p>When using multiprocessing and CUDA, as mentioned <a href="http://pytorch.org/docs/stable/notes/multiprocessing.html#sharing-cuda-tensors">here</a> you have to use start method that is not fork. For example:</p>
<pre><code class="lang-auto">import torch
torch.multiprocessing.set_start_method('spawn')
</code></pre>

username: KelleyYin
cooked: <p>Thanks a lot for the help so far . After adding torch.multiprocessing .set_start_method(“spawn”), there arise a new problem:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test9.py", line 45, in &lt;module&gt;
    torch.multiprocessing.set_start_method('spawn')
  File "/usr/local/lib/python3.5/multiprocessing/context.py", line 231, in set_start_method
    raise RuntimeError('context has already been set')
RuntimeError: context has already been set

</code></pre>
<p>Is there a way around this ?</p>

username: Nabarun_Goswami
cooked: <p>This usually happens if you have not properly wrapped you main in <code>if __name__ == '__main__':</code> construct. Another issue might be that your project has multiple files which have that construct, and these will set the contexts when importing these files.</p>
<ul>
<li>One option is to have only a single entry point in your project which is properly wrapped in that construct,</li>
<li>Or you can call the <code>set_start_method</code> with the argument <code>force=True</code>
</li>
</ul>
<pre><code class="lang-auto">import torch
torch.multiprocessing.set_start_method('spawn', force=True)
</code></pre>
<p>Hope it helps. <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=5" title=":blush:" class="emoji" alt=":blush:"></p>

username: percqdeng
cooked: <p>I had a slightly different problem when training multiple models in multiple gpu in parallel. This is the only answer I found helpful so I hope I can bring it up here.</p>
<p>It can be shown by modifying the example based on the above suggestion. Here I want to run the processes (<code>use_gpu</code>) in all the devices in parallel.</p>
<pre><code class="lang-python">import time
import torch
from torch.multiprocessing import Pool
torch.multiprocessing.set_start_method('spawn', force=True)


def use_gpu(ind):
    t = []
    for i in range(5):
        time.sleep(1)
        a = torch.randn(1000, 1000).cuda(ind)
        t.append(a)
    return t

if __name__ == "__main__":
    # torch.cuda.set_device(3)
    pool = Pool()
    result = []
    a = time.time()
    for i in range(torch.cuda.device_count()):
        result.append(pool.apply_async(use_gpu, (i,)))
    pool.close()
    pool.join()
    print("cost time :", time.time() - a)
</code></pre>
<p>However I couldn’t figure out why I got the error message</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524590031827/work/aten/src/THC/THCTensorRandom.cu line=25 error=46 : all CUDA-capable devices are busy or unavailable
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524590031827/work/aten/src/THC/THCTensorRandom.cu line=25 error=46 : all CUDA-capable devices are busy or unavailable
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524590031827/work/aten/src/THC/THCTensorRandom.cu line=25 error=46 : all CUDA-capable devices are busy or unavailable
</code></pre>
<p>Thanks for the help.</p>

username: calincru
cooked: <aside class="quote no-group" data-username="percqdeng" data-post="5" data-topic="16866">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/percqdeng/48/830_2.png" class="avatar"> percqdeng:</div>
<blockquote>
<p>all CU</p>
</blockquote>
</aside>
<p>I have the same problem. How did you fix it?</p>

username: terryyizhong
cooked: <p>same as you. Have you solved this problem??</p>

username: terryyizhong
cooked: <p>same as you. Have you fix it now?</p>