How to force/manually swap the VRAM into RAM?

GF-Huang: <p>I know that the latest version of PyTorch supports mixed use of VRAM and RAM (when VRAM is insufficient), but in some cases, I want to manually swap VRAM to RAM. Does PyTorch or CUDA provide such APIs?</p>

ptrblck: <p>Could you describe your use case in more detail?<br>
If you want to just move tensors from the GPU to the host, you can use <code>x = x.to("cpu").</code> PyTorch provides APIs for offloading and custom memory allocators via <code>torch.cuda.MemPool</code> in case you are looking for more advanced use cases.</p>

GF-Huang: <p>Sometimes I try run different models in multiple different Jupyter notebooks, they are not popular CNN, ANN, etc.; they are some of my own attempts, which are sensitive to latency. PyTorch/CUDA automatically swaps VRAM to RAM, which is uncontrollable, and I don’t know when the swap will occur, leading to uncontrollable latency.</p>
<p>In details, my computer has 32GB RAM and 8GB VRAM, then I do the follows:</p>
<ol>
<li>Run the <code>model 1</code> in <code>notebook 1</code>, which use about 7GB VRAM.</li>
<li>Run <code>model 2</code> in <code>notebook 2</code>, insufficient VRAM, swapping occurs during <code>model 2</code> running, this will cause delay which is unacceptable for me. (I don’t want to kill the <code>notebook 1</code> jupyter kernel for releasing VRAM, because I want to keep the environment for later use.)</li>
</ol>
<p>What I want?</p>
<ol>
<li>Run the <code>model 1</code> in <code>notebook 1</code>, which use about 7GB VRAM.</li>
<li>Swap <code>notebook 1</code> VRAM into RAM manually.</li>
<li>Run <code>model 2</code> in <code>notebook 2</code>. No swapping so no delay, this is very good.</li>
<li>Swap <code>notebook 2</code> VRAM into RAM manually.</li>
<li>Swap <code>notebook 1</code> RAM into VRAM, run <code>model 1</code>, no swapping no delay, very good.</li>
</ol>

ptrblck: <aside class="quote no-group" data-username="GF-Huang" data-post="3" data-topic="216563">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gf-huang/48/50890_2.png" class="avatar"> GF-Huang:</div>
<blockquote>
<p>PyTorch/CUDA automatically swaps VRAM to RAM, which is uncontrollable, and I don’t know when the swap will occur, leading to uncontrollable latency.</p>
</blockquote>
</aside>
<p>No, it doesn’t. PyTorch will raise an OOM when you are trying to allocate too much memory on the GPU.<br>
I’ve heard some Windows display drivers are automatically swapping GPU memory to the host, but this seems to be a Windows “feature”.</p>

GF-Huang: <p>Yes, I’m Windows 11. I’m sure it will swap into RAM if VRAM insufficient and will not raise OOM.</p>

ptrblck: <p>In that case you are most likely seeing the Windows driver “feature” I mentioned and PyTorch cannot stop it.<br>
If you are on Linux, you will get an OOM unless you explicitly are using UVM, CPU-offloading etc.</p>
<p>EDIT: to be precise: I don’t know if and how this behavior can be disabled, but maybe a Windows expert might know.</p>

Werhows: <p>The first method is adjusting the VRAM allocation in your computer’s UEFI or BIOS. Enter your BIOS and look for an option in the menu named Advanced Features.</p>