Pytorch multiprocess worked much slower and failed because of out of memory

Forceless: <p>Hi, I’m a rookie in parallel.<br>
Recently, I was trying to make my code parallel.<br>
Because of python’s global interpreter lock, I choose to use process parallel.<br>
I tried python’s native lib multiprocess, It raise an error when importing torch.<br>
So I tried pytorch’s multiprocess, it also failed because of some weird errors.<br>
For example, CUDA: Out of Memory, and It shouldn’t happen.<br>
It will occupy tons of memory, and I don’t get it.<br>
I found the logic of multiprocess is also weird.<br>
By the way, when num of subprocesses is low, it won’t fail anymore.<br>
And another tough question is paralleled code is much slower.<br>
Here’s comparison:<br>
8.354568481445312  # paralleled<br>
0.00092315673828125 <span class="hashtag">#unparalleled</span><br>
When I debug, it seems like every subprocess doesn’t only run functions, they run from the start.</p>
<pre><code class="lang-python">from multiprocessing.dummy import freeze_support
from time import time
import torch
from torch import multiprocessing as mp
def f(x):
    x *= torch.randn((3,24,24),device='cuda:0')
if __name__ ==  "__main__":
    freeze_support()# for windows support
    data = torch.randn(16,3,24,24,device='cuda:0')
    mp.set_start_method('spawn')# fail log prompt me to run this
    pool = mp.Pool(mp.cpu_count())
    start = time()
    for i in data:
        pool.apply_async(f,i)
    pool.close()
    pool.join()
    print(time()-start)
    start = time()
    for i in data:
        f(i)
    print(time()-start)
</code></pre>
<p><a href="https://drive.google.com/file/d/1_hlnk8qmYgaQTTu-W-b5IxX7tBEyskfR/view?usp=sharing" rel="noopener nofollow ugc">Full Error Log</a><br>
My Running Environment:<br>
i9-10940X<br>
RTX3090<br>
Ubuntu<br>
Python3.8<br>
torch 1.10.0+cu113<br>
Thank you!</p>

eqy: <p>This kind of parallelization strategy is somewhat unusual as many PyTorch models can be expressed without explicitly relying on multiprocessing/pools (e.g., via parallel dataloaders, distributed data-parallel, etc.). Additionally, in a single GPU setup (as this appears to be), parallelizing CPU processes but ultimately dispatching to the same GPU may increase contention and not actually yield any speedup.</p>
<p>Could you share some more details about the workload/use case that you are trying to speed up?</p>

Forceless: <p>Ok, here’s a slice of my original code</p>
<pre><code class="lang-python">        for key_id, key in enumerate(self.keys):
            pmf_matrix = np.random.choice(
                    self.PerturbationTable, self.shapes[key_id], p=self.prop_table)
            dic[key] *= torch.tensor(pmf_matrix, device=self.device
                                                  )
</code></pre>
<p>As you see, for every layer in a CNN, I dot multiply it with a matrix.<br>
So far, it runs relatively slowly, about 4 iterations per second in resnet18. (RTX3090)<br>
2.5 iterations per second on my laptop (RTX3050).<br>
I hope it could be faster.<br>
Thus I want to speed up it by invoking the parallel technique.<br>
Is my idea of speeding up is wrong?<br>
This code couldn’t speed up anymore when I only have one graphic card.<br>
Beside that, print in my subprocess doesn’t really print anything.</p>

Forceless: <p>My other idea is to let subprocess 1 use GPU, and the other subprocesses use CPU.<br>
But it seems like, calling a subprocess is much wastable.</p>