[Documented fix] Slow execution, Pytorch using GPU shared memory

neoncube: <p>Hi everyone!</p>
<p>I recently encountered slowdowns while training using my consumer-grade GPU (Nvidia GTX), and I wanted to document the root cause.</p>
<p>My GPU has 4GB of dedicated memory and 16GB of shared memory. When training, I would encounter an issue where training was fast at first and then would slow down massively. This wasn’t super obvious: it just felt like training was slow, and I didn’t know why.</p>
<p>The issue turned out to be that Pytorch doesn’t differentiate between GPU dedicated memory and GPU shared memory, but accessing shared GPU memory is, of course, much slower than accessing dedicated GPU memory. In a training loop, tensors end up first getting allocated in dedicated memory, then getting allocated in shared memory once the dedicated memory is full. When both dedicated and shared memory is full, Pytorch flushes its memory cache and the cycle starts over again.</p>
<p>My testing showed that when Pytorch is allocating tensors in shared memory, my training loop takes about 3x the time it takes when the tensors are allocated in dedicated memory (4.5s vs 1.5s).</p>
<p>A good way to prevent this is to tell NVIDIA to restrict Pytorch to only use dedicated GPU memory.</p>
<p>On Windows, there are two ways to do this:</p>
<ol>
<li>Make it so that <em>all</em> programs can’t access shared memory. To do this, open NVIDIA Control Panel, go to <code>Manage 3D settings</code>, and on the <code>Global Settings</code> tab, change <code>CUDA - Sysmen Fallback Policy</code> from <code>Driver Default</code> to <code>Prefer No Sysmem Fallback</code>, and click the <code>Apply</code> button when it appears, several seconds later.</li>
<li>Make it so that only Python can’t access shared memory. To do this, open NVIDIA Control Panel, go to <code>Manage 3D settings</code>, click the <code>Program Settings</code> tab, click <code>Add</code>, add <code>Python3.exe</code> (or <code>Python.exe</code>?), and then follow the same instructions as above. NVIDIA also has a nice support article with how to do this, including screenshots: <a href="https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/system-memory-fallback-for-stable-diffusion" class="inline-onebox" rel="noopener nofollow ugc">System Memory Fallback for Stable Diffusion | NVIDIA</a></li>
</ol>
<p>The steps should be similar on Linux. See <a href="https://hatchjs.com/nvidia-control-panel-linux/" class="inline-onebox" rel="noopener nofollow ugc">How to Install and Use the NVIDIA Control Panel on Linux</a> for launching the Nvidia Control Panel on Linux</p>
<p>After enabling this setting, your training should be restricted to using only your GPU’s dedicated memory!</p>
<p>I hope this helps someone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

neoncube: <p>My guess is that server-grade GPUs don’t have GPU shared memory, so this probably only affects consumer-grade GPUs. Even so,I think it might be useful to document this somewhere in the official documentation.</p>
<p>Or, ideally, Pytorch would be able to tell the difference between dedicated GPU memory and shared GPU memory and would flush the cache whenever allocating a tensor would result in it being allocated in shared memory! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <aside class="quote no-group" data-username="neoncube" data-post="1" data-topic="218909">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neoncube/48/76417_2.png" class="avatar"> neoncube:</div>
<blockquote>
<p>The issue turned out to be that Pytorch doesn’t differentiate between GPU dedicated memory and GPU shared memory, but accessing shared GPU memory is, of course, much slower than accessing dedicated GPU memory. In a training loop, tensors end up first getting allocated in dedicated memory, then getting allocated in shared memory once the dedicated memory is full. When both dedicated and shared memory is full, Pytorch flushes its memory cache and the cycle starts over again.</p>
</blockquote>
</aside>
<p>I assume you mean system memory for offloading. If so then note that PyTorch does not trigger this behavior and it’s caused by the Windows driver as a “feature”.</p>
<aside class="quote no-group" data-username="neoncube" data-post="1" data-topic="218909">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neoncube/48/76417_2.png" class="avatar"> neoncube:</div>
<blockquote>
<p>The steps should be similar on Linux.</p>
</blockquote>
</aside>
<p>No, since the Linux driver does not offload to the host. On Linux you could use the <code>torch.cuda.MemPool</code> API to e.g. use UVM to offload (sections) of your tensors but would also see the same poor performance for these sections of your code. Alternatively, you could use the native offloading APIs in PyTorch.</p>
<aside class="quote no-group" data-username="neoncube" data-post="2" data-topic="218909">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neoncube/48/76417_2.png" class="avatar"> neoncube:</div>
<blockquote>
<p>Even so,I think it might be useful to document this somewhere in the official documentation.</p>
</blockquote>
</aside>
<p>PyTorch does not trigger this behavior but some Windows drivers. I would assume this is already documented somewhere (I’m not using Windows so don’t know).</p>

neoncube: <aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="218909">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I assume you mean system memory for offloading. If so then note that PyTorch does not trigger this behavior and it’s caused by the Windows driver as a “feature”.</p>
</blockquote>
</aside>
<p>Yes, I know, thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">   Still, I think it might be worth looking into whether Pytorch could automatically detect the GPU physical memory limit and keep the cache beneath that limit when possible. If people agree, then I can file on bug on the Pytorch Github <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group quote-modified" data-username="ptrblck" data-post="3" data-topic="218909">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I would assume this is already documented somewhere.</p>
</blockquote>
</aside>
<p>I couldn’t find anything in the Pytorch documentation, and I think this is an issue that essentially every Pytorch Windows user with a consumer-grade GPU will run into <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="218909">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>No, since the Linux driver does not offload to the host.</p>
</blockquote>
</aside>
<p>Ah, interesting! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>If your moderator powers permit, would you mind moving this to the Windows category, since it seems like it’s only valid on Windows?</p>

ptrblck: <aside class="quote no-group" data-username="neoncube" data-post="4" data-topic="218909">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neoncube/48/76417_2.png" class="avatar"> neoncube:</div>
<blockquote>
<p>Still, I think it might be worth looking into whether Pytorch could automatically detect the GPU physical memory limit and keep the cache beneath that limit when possible.</p>
</blockquote>
</aside>
<p>No, I don’t think changing PyTorch’s default behavior makes sense since PyTorch itself will properly raise an OOM when running out of memory on the device. If you want to lower the available memory to your PyTorch process you can use <a href="https://pytorch.org/docs/stable/generated/torch.cuda.set_per_process_memory_fraction.html#torch-cuda-set-per-process-memory-fraction"><code>torch.cuda.set_per_process_memory_fraction</code></a> which should hopefully also avoid this behavior on your Windows system.</p>
<aside class="quote no-group" data-username="neoncube" data-post="4" data-topic="218909">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neoncube/48/76417_2.png" class="avatar"> neoncube:</div>
<blockquote>
<p>and I think this is an issue that essentially every Pytorch Windows user with a consumer-grade GPU will run into <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>I don’t think all Windows drivers enabled this behavior by default, but again I don’t use Windows and don’t know how widespread this is. I believe this is the second time a user mentioned this as an issue, so either Windows users expect this behavior or don’t think it’s hostile (I do think it is unexpected).</p>

neoncube: <p><a href="https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/system-memory-fallback-for-stable-diffusion" rel="noopener nofollow ugc">This article from Nvidia</a> says that it’s enabled in driver 536.40+, which was released in June 2023.</p>