Parallelize simple for-loop for single GPU

username: jose
cooked: <p>Hello,</p>
<p>I have a for loop which makes independent calls to a certain function. The calls should be processed in parallel, as they are completely independent. I have the following code which works for CPU. For GPU I am still trying to get it working.</p>
<pre><code class="lang-auto">import multiprocessing
from joblib import Parallel, delayed

class some_function(Function):
    @staticmethod
    def forward(ctx, x):
        pass # here goes the code of the forward pass

    @staticmethod
    def backward(ctx, grad_output):
        pass

class L_Field(nn.Module):
    def __init__(self):
        super(L_Field, self).__init__()

    def forward(self, x):
        batch_size, channels, height, width = x.shape
        num_cores = multiprocessing.cpu_count()
        results = Parallel(n_jobs=num_cores, backend="threading")(delayed(some_function.apply)(x[b, c]) for b in range(batch_size) for c in range(channels))
        out = torch.stack(results).reshape(batch_size, channels, height, width)
        return out
</code></pre>
<p>For the GPU, I tried the following:</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp

class some_function(Function):
    @staticmethod
    def forward(ctx, x):
        pass # here goes the code of the forward pass

    @staticmethod
    def backward(ctx, grad_output):
        pass

class L_Field(nn.Module):
    def __init__(self):
        super(L_Field, self).__init__()

    def forward(self, x):
        batch_size, channels, height, width = x.shape
        num_cores = mp.cpu_count()
        x.share_memory_()
        pool = mp.Pool(processes=num_cores)
        results = [pool.apply(some_function.apply, args=(x[b, c],)) for b in range(batch_size) for c in range(channels)]
        pool.close()
        pool.join()
        out = torch.stack(results).reshape(batch_size, channels, height, width)
        return out
</code></pre>
<p>For which I get:</p>
<pre><code class="lang-auto">RuntimeError: Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).
</code></pre>
<p>If I do the detach:</p>
<pre><code class="lang-auto">RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
</code></pre>
<p>Then, if I do <code>mp.set_start_method('spawn', force=True)</code>:</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: out of memory
</code></pre>
<p>Any ideas will be appreciated.</p>

username: ismaelardo
cooked: <p>could you solve it?<br>
i have the same problem, i want to do a multiprocess over a for loop, here is my forward code:</p>
<pre><code class="lang-auto">def forward(self,x):
    output,(self.h,self.c)=self.lstm(x,(self.h,self.c))
    i=0
    out=[]
    for batch_size in output.batch_sizes:
      out.append(self.linear(output.data[i:batch_size+i,:]))
      i=i+batch_size
      t=t+1
    outs=torch.cat(out)
    return outs```
(output is a PaquedSequence and self.lstm and self.linear are the respective nn.Modules)
i have tried differents ways but allways i receive the ```
RuntimeError: Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).
</code></pre>
<p>is it possible to do multiprocess over inputs that are non-leaf and requires grad?<br>
thank you !</p>

username: ismaelardo
cooked: <p>Please, <span class="mention">@ptrblck_de</span> (or some expert),  it is possible to make multiprocessing over inputs that require grad an are not leafs?, how it is done?<br>
I would appreciate any help.</p>

username: derek_kim
cooked: <p>I have the same issue.</p>

username: konstantinosKokos
cooked: <p>also waiting for an update on this issue</p>

username: Dongsu_Zhang
cooked: <p>I also have the same issue <img src="https://discuss.pytorch.org/images/emoji/apple/tired_face.png?v=9" title=":tired_face:" class="emoji" alt=":tired_face:"></p>

username: Gaoyuan_Yang
cooked: <p>Hi, did you find a solution here, I find the similar question <a href="https://discuss.pytorch.org/t/how-to-parallelize-a-loop-over-the-samples-of-a-batch/32698">https://discuss.pytorch.org/t/how-to-parallelize-a-loop-over-the-samples-of-a-batch/32698</a> and <a href="https://discuss.pytorch.org/t/running-multiple-modules-in-parallel/58164/2">https://discuss.pytorch.org/t/running-multiple-modules-in-parallel/58164/2</a>, both answers give a way of reformulate the problem, however they did not solve the problem of multi-processing inside one GPU inside forward function.</p>