RAM explodes when using SharedMemory and CUDA

username: lkc
cooked: <p>I would like to use multiprocessing to launch multiple training instances on CUDA device. Since the data is common between the processes, I want to avoid data copy for every process. I’m using python 3.8’s <code>SharedMemory</code> from multiprocessing module to achieve this.</p>
<p>I can allocate a memory block using SharedMemory and create as many processes as I’d like with constant memory (RAM) usage. However, when I try to send tensors to CUDA, the memory scales linearly with the number of processes. It appears as if when <code>c.to(device)</code> is called, the base data is copied for every process.</p>
<p>Does any one know why this is happening? Any ideas to mitigate this issue?</p>
<p>Here is the sample code I’m using:</p>
<pre><code class="lang-auto">import numpy as np
from multiprocessing import shared_memory, get_context
import time
import torch
import copy

dim = 10000
batch_size = 10
sleep_time = 2
npe = 1  # number of parallel executions

# cuda
if torch.cuda.is_available():
    dev = 'cuda:0'
else:
    dev = "cpu"
device = torch.device(dev)


def step(i, shr_name):
    existing_shm = shared_memory.SharedMemory(name=shr_name)
    np_arr = np.ndarray((dim, dim), dtype=np.float32, buffer=existing_shm.buf)
    b = np_arr[i * batch_size: (i + 1) * batch_size, :]
    b = torch.Tensor(b)

    # This is just to explicitly copy the tensor so that it has nothing to do
    # with the shared memory block
    c = copy.deepcopy(b)
    
    # If tensor c is sent to the cuda device, then RAM scales linearly
    # with the number of parallel executions.
    # If c is not sent to cuda device, memory consumption is constant.
    c = c.to(device)
    
    time.sleep(sleep_time)
    existing_shm.close()


def create_shared_block():
    a = np.random.random((dim, dim)).astype(np.float32)
    shm = shared_memory.SharedMemory(create=True, size=a.nbytes, name='sha')
    np_arr = np.ndarray(a.shape, dtype=np.float32, buffer=shm.buf)
    np_arr[:] = a[:]
    return shm, np_arr


if __name__ == '__main__':
    # create shared memory block
    shm, np_arr = create_shared_block()
    # create list of inputs to be executed in parallel
    inp = [[x, 'sha'] for x in range(npe)]
    print(inp)

    # sleep added before and after launching multiprocessing to monitor the memory consumption
    print('before pool')  # to check memory with top or htop
    time.sleep(sleep_time)
    
    context = get_context('spawn')
    with context.Pool(npe) as pool:
        print('after pool')  # to check memory with top or htop
        time.sleep(sleep_time)
        
        pool.starmap(step, inp)

    time.sleep(sleep_time)

    shm.close()
    shm.unlink()

</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>The reason this happens is that the cuda driver needs the data to be in a specific type of RAM (pinned memory) to be sent on the GPU. So it makes a temporary copy if it is not in the right place.<br>
I guess this is why you see extra RAM usage when you send your Tensors on the GPU.</p>

username: lkc
cooked: <p>That’s what I was wondering too. But what I’m puzzled about is - I’m only sending a deepcopy of a slice from a big tensor to the gpu. So even if the new <em>much smaller</em> tensor is copied to the pinned memory, memory shouldn’t explode. However, it appears the big tensor is also sent to the pinned memory even though it is not sent to the GPU.</p>
<p>Is there any way to use same data on GPU across multiple processes?</p>
<p>Thanks!</p>

username: albanD
cooked: <aside class="quote no-group" data-username="lkc" data-post="3" data-topic="97422">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lkc/48/28917_2.png" class="avatar"> lkc:</div>
<blockquote>
<p>Is there any way to use same data on GPU across multiple processes?</p>
</blockquote>
</aside>
<p>You can check the doc about this: <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing" class="inline-onebox">Multiprocessing best practices — PyTorch 2.1 documentation</a></p>
<p>Not sure about the memory… It does the same thing if you do <code>c = b.clone()</code> (I’m not 100% sure what deepcopy is doing).</p>

username: lkc
cooked: <p>Yeah, I did check that page. I’m not able to find anything about sharing the same data.</p>
<p><code>deepcopy</code> will copy the data in the tensor and not just the reference. I guess <code>clone</code> has the graph information which <code>deepcopy</code> does not.  I’ll poke around further, and will post back if I find any alternative. Thanks</p>