Allocate cuda tensor in subprocess

florin: <p>Here’s another <code>torch.multiprocessing</code> situation I can’t wrap my head around. This example is working:</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp

def put_in_q():
    x = torch.IntTensor(2, 2).fill_(22)
    x = x.cuda()
    print(x)

p = mp.Process(target=put_in_q, args=())
p.start()
p.join()
</code></pre>
<p>But not this one:</p>
<pre><code class="lang-auto">import torch
import multiprocessing as mp

class CudaProcess(mp.Process):
    def __init__(self):
        mp.Process.__init__(self)

    def run(self):
        x = torch.IntTensor(2, 2).fill_(22)
        x = x.cuda()
        print(x)

p = CudaProcess()
p.start()
p.join()
</code></pre>
<p>The error I’m getting is:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'THException'
  what():  cuda runtime error (3) : initialization error at .../pytorch/torch/lib/THC/THCGeneral.c:70
</code></pre>

apaszke: <p>CUDA multiprocessing is quite complicated, and I wouldn’t recommend it, unless it would give you huge performance benefits. The two most important things to remember are:</p>
<ol>
<li>You can’t use the default <code>fork</code> start method. You need to switch multiprocessing to either use <code>spawn</code> (simpler) or <code>forkserver</code> (possible more efficient, but still subtle - server needs to be initialized before CUDA is initialized). The problem is that CUDA doesn’t support forking the process once the context has been initialized. The child will be in an inconistent state and it’s UB.</li>
<li>A shared CUDA storage/tensor from another process <strong>mustn’t go out of scope in the process that created it</strong>, for at least as long as it’s used in other processes. Otherwise, UB again.</li>
</ol>

kmichaelkills: <p>I’m not familiar with all the architecture in pytorch, but I would like to share some thoughts if I may. Wouldnt multi-threading mult-gpu work if:</p>
<ol>
<li>C++ part release GIL systematically</li>
<li>Memory (de)allocation isnt blocking (use memory pool like Tensorflow instead of CachedAllocation)</li>
</ol>
<p>I guess moving all c++ part to cffi would solve 1)  and would make it even possible to use pypy (lua torch  used JIT but in python its suddenly a bad idea?)</p>

florin: <p>Thank you for the quick reply. For some context, I am trying to launch several agents inheriting <code>mp.Process</code> playing in separate environments, each enqueuing frames in a shared <code>Queue</code>. A single prediction subprocess is dequeuing batches of frames, does an inference step through the neural network estimator and passes back actions to each agent.</p>
<p>I am already seeing nice speed-ups compared to the naive asynchronous implementation and after I changed the start method to <code>spawn</code> it seemed to be running fine on my laptop with <code>cuda</code> support (shaving about 20% of the execution time when using <code>cuda</code>). However when I tested it on our servers I got a <code>RuntimeError: CUDNN_STATUS_NOT_INITIALIZED</code>.</p>
<p>I get now it’s not really trivial setting this up and I will continue working on the implementation leaving <code>cuda</code> aside for now. I do plan to revisit the issue and I’ll post here about any developments.</p>

florin: <p><a class="mention" href="/u/apaszke">@apaszke</a> also, how is multi-gpu support being implemented? I’ve seen the Hogwild! example but it’s cpu only.</p>
<p>edit:</p>
<blockquote>
<p>The child will be in an inconistent state and it’s UB</p>
</blockquote>
<p>What is UB? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

kmichaelkills: <p>UB=undefined behaviour I guess</p>

apaszke: <p>Yeah, UB is undefined behaviour = anything could happen now.</p>
<p><a class="mention" href="/u/kmichaelkills">@kmichaelkills</a> we do release GIL in C++ parts and memory (de)allocation isn’t blocking. You can think of the caching allocator like a self expanding memory pool.</p>
<p>Multi-threaded multi-GPU works, and is used to dispatch kernels by our <code>DataParallel</code> module and <code>data_parallel</code> function. Still, even though we carefully release the GIL as soon as we enter C++, the contention on the lock, that serializes execution, makes it nearly impossible to saturate 8 modern GPUs. That’s why we’ve started exploring CUDA multiprocessing.</p>
<p>Regarding cffi, we didn’t want to add more dependencies, and having more fine grained control over GIL and some other aspects makes it easier for us to develop the C code. I’d love to support PyPy, but there are very few people using it, so it’s not really a top priority for us.</p>
<p><a class="mention" href="/u/florin">@florin</a> well, the frames are on the CPU already, right? Can’t you cat them there and ship the whole batch to CUDA before giving it to the network? Why do you need CUDA multiprocessing there?</p>
<p>I don’t really understand the thing about multi-GPU. You can switch between GPUs using <code>torch.cuda.device</code>, and that’s all you need to execute code on another device. Alternatively, if you want to have purely asynchronous training, you could start up some threads or processes, have them run on separate GPUs, and sync parameters periodically. This is a use case where CUDA multiprocessing can come in handy, because the shared memory never goes out of scope.</p>