What is the shared memory?

username: ysleer
cooked: <p>hi</p>
<p>I am trying to train a model using multiprocessing.</p>
<p>In the example below (<a href="https://pytorch.org/docs/1.6.0/notes/multiprocessing.html?highlight=multiprocessing" class="inline-onebox" rel="noopener nofollow ugc">Multiprocessing best practices — PyTorch 1.6.0 documentation</a>), model.share_memory() is used.</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp
from model import MyModel

def train(model):
    # Construct data_loader, optimizer, etc.
    for data, labels in data_loader:
        optimizer.zero_grad()
        loss_fn(model(data), labels).backward()
        optimizer.step()  # This will update the shared parameters

if __name__ == '__main__':
    num_processes = 4
    model = MyModel()
    # NOTE: this is required for the ``fork`` method to work
    model.share_memory()
    processes = []
    for rank in range(num_processes):
        p = mp.Process(target=train, args=(model,))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()
</code></pre>
<p>What exactly does shared memory mean??</p>
<p>Is a model created for each process??</p>
<p>Or is it that each process uses one model shared??</p>
<p>I don’t really understand what is in the document (<a href="https://www.geeksforgeeks.org/ipc-shared-memory/" class="inline-onebox" rel="noopener nofollow ugc">IPC through shared memory - GeeksforGeeks</a>).</p>
<p>thank you.</p>

username: ptrblck
cooked: <p>The <a href="https://en.wikipedia.org/wiki/Shared_memory">Wikipedia article</a> explains shared memory maybe a bit easier to understand.<br>
It’s basically a memory pool, which can be used by multiple processes to exchange information and data.</p>

username: DanielWang
cooked: <p>Is there any Pytorch Extention that supports GPU-based shared memory on CUDA?</p>

username: ptrblck
cooked: <p>I’m not sure if I understand the question correctly, but you can directly use shared memory in your custom CUDA kernels called in the extension.</p>

username: DanielWang
cooked: <p>Does this mean I can call</p>
<pre><code class="lang-auto">mode.shared_memory(); 
model.to(device); 

p1 = create_process()
p2 = create_process()

p1.model_train(model)
p2.model_train(model)

</code></pre>

username: ptrblck
cooked: <p>No, the kernel-level shared memory is not the system shared memory used for IPC.<br>
The former can be used in CUDA code as described <a href="https://developer.nvidia.com/blog/using-shared-memory-cuda-cc/">here</a>.</p>

username: tengerye
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> So the <code>model.share_memory()</code> enables the cached model to be shared across different processes in a single GPU instead of multiple GPUs?</p>
<pre><code class="lang-auto">model1.to(device1)
model1.shared_memory()

p1 = create_process()
p2 = create_process()

p1.model_train(model1)
p2.model_train(model1)

model2= model1.clone().to(device2)
model2.shared_memory()

p3 = create_process()
p4 = create_process()

p3.model_train(model2)
p4.model_train(model2)
</code></pre>
<p>May I ask if the above snippet works fine?</p>

username: ptrblck
cooked: <p><code>tensor.share_memory_()</code> will move the tensor data to shared memory on the host so that it can be shared between multiple processes. It is a no-op for CUDA tensors as described <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.share_memory_.html#torch.Tensor.share_memory_">in the docs</a>. I don’t quite understand the “in a single GPU instead of multiple GPUs” as this type of shared memory is not used on the GPU (i.e. it’s not the CUDA kernel-level shared memory).</p>

username: tengerye
cooked: <p>I am still a little confused. Let’s say I have 2 GPUs and would like to run 2 processes on each GPU. All 4 of the processes are independent.</p>
<p>May I ask some questions please:</p>
<ol>
<li>There is no need to use <code>share_memory()</code>, right?</li>
<li>I just need to clone 2 models for each GPU and run multiprocessing of pytorch. Is that correct?</li>
<li>Is there anything else I need to do?</li>
</ol>

username: ptrblck
cooked: <p>If all processes are independent, e.g. each process is training an independent model and is not using model sharding, data parallel etc. then you should just launch your processes on the desired device. Since GPU resources will be shared between processes, you would most likely see a slowdown compared to a single process using a single GPU.</p>

username: tengerye
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi, thank you for your kind reply. In my case, I need to calculate the Hessian matrix of loss w.r.t the weights of model for <strong>each data point</strong> (using <code>torch.autograd.grad()</code>). There is no need update weights. May I ask do I need to explicitly clone the models for each processes to make sure the gradients will not be shared?</p>

username: JuyiLin
cooked: <p>If you don’t want to share gradient, you need <code>.detach()</code></p>

username: SKTheLearner
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I am beginning my transition from TensorFlow to PyTorch!!<br>
have a similar GPU question.<br>
I am under Windows 10 with NVIDIA RTX A2000 GPU(4GB) with 64GB RAM(50% goes to shared memory-VRAM)…so effectively I expected 36(32+4) GB GPU memory in total.</p>
<ol>
<li>When I just create tensors and commit to ‘cuda’, I can see it using shared memory also…</li>
<li>Then I created a model, committed to GPU , so my data as well as model in GPU,but it doesnt seem to use shared memory of 36GB while training the model…</li>
</ol>
<p>using a Conv3d model, input is a 3d volume (1,256,256,128) of unint8 …training batch_size=2</p>
<p>Does PyTorch automatically use the shared memory, if you commit your model as wells as data to cuda? Is there anything I need to do, apart from using model.to as well tensor.to push every thing to GPU ?</p>
<p><strong><strong>ERROR</strong></strong>**<br>
return F.conv3d(<br>
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 15.51 GiB. GPU 0 has a total capacty of 4.00 GiB of which 0 bytes is free. Of the allocated memory 4.25 GiB is allocated by PyTorch, and 1.95 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>

username: ptrblck
cooked: <p>Could you describe what exactly you mean by “shared memory”?<br>
Are you assuming the GPU will use its device memory as well as the host memory as unified memory?<br>
If so, that’s not the case and you would need to use the device memory only (4GB) for your model parameters as well as GPU computation.<br>
You could try to offload tensors to the CPU, but you would pay for the transfer each time.</p>

username: SKTheLearner
cooked: <p>to start with …my bad…PyTorch model/data  is using dedicated GPU memory as well as shared memory under Windows 10(and am sure it will 10+)…</p>
<p>On to Shared Memory , most of the intel laptops have onboard video/GPU chip…but they dont have anything called ‘memory’(we call it VRAM…Video Random…)…so they depend on system RAM… under Windows, by default, 50% of your RAM is reserved for it…ie, if you have 16GB RAM, 8 GB is reserved for VRAM…so 50%…get to the task manager under windows-&gt;performance…GPU…</p>
<p>So if you have dedicated GPU with dedicated memory of 4 GB and you have windows laptop with 64 GB RAM…your effective GPU memory available is (32GB+4GB)…36 GB which you should be able to use for your model/data…I can confirm PyTorch uses it… on the other hand , TensorFlow doesnt support GPU under windows after v 2.10…(hence my transition to PyTorch, as an aside)</p>

username: ptrblck
cooked: <p>PyTorch doesn’t perform any of this offloading and I would be careful calling it shared memory as it usually refers to the shared memory in a CUDA kernel when talking about GPUs.<br>
In any case, this might be a Windows-specific driver feature, which starts with page offloading once a threshold is met, but I’m not using Windows so don’t know if and when it happens.<br>
You should however note that it will come with a large performance penalty.</p>

username: SKTheLearner
cooked: <p>yes…50% thingy is Windows specific…my quest was to use that available memory to train on GPU(RAM,VRAM whatever…there is resource…so should be able to use it…) maybe a hit on performance… but for mere mortals like me cant afford an Nvidia A100(with 40-80GB memory)…, PyTorch is looking good with my computing resource as of now…and looks to be right up my ally…control on every single line of code you write…</p>
<p>here we go…PyTorch Model/Dataset …<br>
1.BEFORE:::<a href="https://drive.google.com/file/d/1yoYUdCipujtE2hVd9XKQc3f4nRzHEFxS/view?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Before.JPG - Google Drive</a><br>
2.Model Training Started: <a href="https://drive.google.com/file/d/1U65WN5pXaEw3YV6F2_e2MnBWN3J09SwY/view?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">model_running.JPG - Google Drive</a></p>