Why PyTorch used so many CPU RAM?

Hao-Yan: <p>I did some experiment with below code:</p>
<pre><code class="lang-auto">import torch
cpu, gpu = torch.device('cpu'), torch.device('cuda:0')
size = (1, 1, 1)
t_gpu1 = torch.zeros(size, device=gpu)
</code></pre>
<p>With command <code>pmap -x pid</code>, I see that there is a large <code>anon</code> mapping memory(about 1GB), which is non file-backend memory and most of it is in <code>RSS</code>.</p>
<pre><code class="lang-auto">000055d6b008d000 1172860 1171972 1171972 rw---   [ anon ]
</code></pre>
<p>What’s more, from the output of <code>cat /proc/pid/smaps</code>, seems this large memory is <strong>private</strong>, which means it can not be shared between multiple process. And it is <strong>dirty</strong>, which means it need to be written out to swap area so it can be recovered when necessary(I don’t know why it is <code>Dirty</code>?).</p>
<pre><code class="lang-auto">VmFlags: rd wr mr mw me ac sd 
55d6b008d000-55d6f79ec000 rw-p 00000000 00:00 0                          [heap]
Size:            1172860 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:             1171972 kB
Pss:             1171972 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:   1171972 kB
Referenced:      1171336 kB
Anonymous:       1171972 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
</code></pre>
<p>Finally I try to use <code>cuda-gdb</code> to attach this python process and dump the memory at this address:</p>
<pre><code class="lang-auto">(cuda-gdb) dump binary memory pytorch-mem.bin 0x000055d6b008d000 0x000055d6b008d000+1172860000
(cuda-gdb) shell cat pytorch-mem.cat | strings &gt; pytorch-mem.bin.strings 
</code></pre>
<p>I see lots of kernel code symbols in this memory area as below, but  <strong>why most of them are loaded into memory with so simple code run, and since this memory is Private, which means it can not be shared across multiple process, so when multiple process  run at the same time, each of them will take this much memory</strong> ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://user-images.githubusercontent.com/6327453/76579550-e431fd80-6489-11ea-9141-f4e0eb3eb241.png" title="image" rel="nofollow noopener"><img src="https://user-images.githubusercontent.com/6327453/76579550-e431fd80-6489-11ea-9141-f4e0eb3eb241.png" alt="image" width="690" height="456"><div class="meta">
<svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2510×1662</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg>
</div></a></div></p>
<p>And I check the size of <code>libtorch.so</code>, actually it is about <code>1.2G</code>, which should contain both cpu code and gpu(kernel) code. Below is the file-backend mapping of the <code>libtorch.so</code></p>
<pre><code class="lang-auto">7f7c896f7000-7f7ccced2000 r-xp 00000000 08:02 38546131                   /usr/share/miniconda3/envs/venv/lib/python3.7/site-packages/torch/lib/libtorch.so
Size:            1105772 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:              304636 kB
Pss:              304636 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:    304636 kB
Private_Dirty:         0 kB
Referenced:       304636 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:		0
VmFlags: rd ex mr mw me sd
</code></pre>
<p>It is <code>Private</code> memory mapping, which also a little surprises me, <strong>will this mean it can not be shared across multiple process even for the CPU code?</strong></p>
<p>For the large <code>anon</code> mapping which contains the kernel code, my guess is that maybe cuda driver allocate that memory to store those gpu code, and those area is mapped to gpu’s virtual address space thus the gpu core can directly access and execute it. <strong>Does anyone know more detail here?</strong></p>

Hao-Yan: <p>According to <a href="https://unix.stackexchange.com/questions/478533/why-are-program-images-and-shared-libraries-considered-private-to-a-process" rel="nofollow noopener">https://unix.stackexchange.com/questions/478533/why-are-program-images-and-shared-libraries-considered-private-to-a-process</a>, the <code>Private</code> pages is <code>copy-on-write</code>, if it is only readable, it still can be shared by the kernel in physical memory. But the <code>anon</code> part is still a issue, because now it is <code>Dirty</code>.</p>

albanD: <p>Hi,</p>
<p>I don’t think there is anything we can do about this. The cuda driver loads all the code this way <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
I can think of two reasons for this: one is to have all the code readily available and that can be easily sent to the GPU. The other one is to be able to do unified memory.</p>

Hao-Yan: <p>Thanks for your quick response.</p>
<p>Is it possible for PyTorch to use CUDA Driver API instead of CUDA Runtime API, thus PyTorch should have more control over how to load the kernel code.</p>
<p>This issue becomes a bottleneck when we try to deploy our product with multiple process, for 14 process, it takes 85%+ memory of 32G. It is ok for train, but when used it in production scenarios, it is an issue.</p>

albanD: <p>Maybe <a class="mention" href="/u/ptrblck">@ptrblck</a> has a better idea of how feasible that is?</p>

ptrblck: <p>I’m not sure that the Driver API would solve your problem and would need some more information about your use case and what you are trying to achieve with these 14 processes.</p>
<p>I.e. is each process loading a different model or script or are these just replica on a single GPU?</p>

Hao-Yan: <p>Our case is that we have 2 GPUs, created 7 process for each GPU to deal with different inputs. Each process will load the same model into GPU(I know we can optimize here), our initial thinking is that GPU RAM should be the bottleneck, but reality is that CPU becomes the bottleneck first, that’s the reason why I did the investigation and seek help here.</p>

ptrblck: <aside class="quote no-group" data-username="Hao-Yan" data-post="7" data-topic="73151">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hao-yan/48/41004_2.png" class="avatar"> Hao-Yan:</div>
<blockquote>
<p>created 7 process for each GPU to deal with different inputs</p>
</blockquote>
</aside>
<p>I’m not sure why you need a dedicated process for different inputs.<br>
Could you explain a bit more, in which way the inputs differ?</p>
<aside class="quote no-group" data-username="Hao-Yan" data-post="7" data-topic="73151">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hao-yan/48/41004_2.png" class="avatar"> Hao-Yan:</div>
<blockquote>
<p>our initial thinking is that GPU RAM should be the bottleneck, but reality is that CPU becomes the bottleneck first</p>
</blockquote>
</aside>
<p>That might be the case, e.g. if your data loading is not fast enough.<br>
However, do you see that multiple processes (including recreating the model) on a single device speed things up?<br>
I would assume that multiple workers in a single <code>DataLoader</code> would give a better performance.</p>

Hao-Yan: <p>The inputs are different video streams, sorry, I can’t talk about more detail here.</p>
<p>Our current arch is each process deal with each video stream separately, and we have MPS enabled.  Another arch is to only use one process and batch all inputs together. One bottleneck here is the GPU memory for large batch size. So you think the latter one will have better performance? (Let’s just assume after all model initialized)</p>

chandrachud: <p>Hi, have you been able to find any solution? Currently struggling with CPU RAM saturation myself. It’s awful that CPU RAM stays full even after moving models to GPU.</p>

Erdi: <p>Is there any update on this issue? I had the same issue, which makes impossible to train any model…</p>

Alejandro_Castaneira: <p>I confirm that I am also having the same issue, it is becoming very difficult to deploy models to production across multiple inference workers as CPU RAM is running out fast.</p>