How to allocate more GPU memory to be reserved by PyTorch to avoid "RuntimeError: CUDA out of memory"?

HitM: <p>Hello,<br>
I’m not experienced in PyTorch very well and perhaps asking a weird question.</p>
<p>I’m running my PyTorch script in a docker container and I’m using GPU that has 48 GB.<br>
Although it has a larger capacity, somehow PyTorch is only using smaller than 10GiB and causing the “CUDA out of memory” error.</p>
<p>Is there any method to let PyTorch use more GPU resources available?<br>
I know I can decrease the batch size to avoid this issue, though I’m feeling it’s strange that PyTorch can’t reserve more memory, given that there’s plenty size of GPU.</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 124.00 MiB (GPU 0; 47.51 GiB total capacity; 9.21 GiB already allocated; 15.06 MiB free; 9.74 GiB reserved in total by PyTorch)
</code></pre>
<p>Thank you in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

HitM: <p>And I noticed that even after reducing the batch size, I still encounter the problem at the second epoch (not while the first mini-batch iteration).<br>
And I confirmed that I’m not trying to save the list of models or the list of losses themselves. (Saving loss.item() as a list instead)</p>
<p>Is it probably because the accumulated gradient graph is exploding in size and it’s unavoidable?</p>
<p>I tried <code>torch.cuda.empty_cache()</code> too but it didn’t help in my case.</p>

Andrei_Cristea: <p>Does the issue go away if you run the same process on the machine directly (as opposed to running it inside the Docker container?) In that case, it could be due to some kind of Docker (or Docker + CUDA) limitation. I’ve run into this before and couldn’t find an easy fix online, but you might be well served to also ask the same question on a Docker forum.</p>
<p>Regardless, if the batch size is part of what’s hogging memory, you could give <a href="https://github.com/rentruewang/koila" rel="noopener nofollow ugc">this project</a> a try. With 1.6k stars on Github, it looks promising, though I cannot confirm whether it works.</p>

ptrblck: <p>Could you check if other processes are using the device and are allocating memory via <code>nvidia-smi</code>?<br>
Also, make sure that <code>torch.cuda.set_per_process_memory_fraction(fraction, device=None)</code> is not set somewhere in your script.</p>

HitM: <p>Hi Andrei,</p>
<p>Thank you for your comment, actually, I have been avoiding running the code on the machine directly and I don’t have an equivalent environment that I have inside the container.<br>
But yes, it seems to be a Docker problem and I may not understand how the container allocates and limits the memory usage correctly.<br>
I will look into the provided link too, thank you!</p>

HitM: <p>Hi ptrblck,<br>
Thank you for your comment.<br>
I’m not setting <code>torch.cuda.set_per_process_memory_fraction(fraction, device=None)</code> in my script.<br>
And here’s the result of nvidia-smi:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3ee231b3cbd63d5af8603d624e59a2d0238a2a9e.png" data-download-href="https://discuss.pytorch.org/uploads/default/3ee231b3cbd63d5af8603d624e59a2d0238a2a9e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3ee231b3cbd63d5af8603d624e59a2d0238a2a9e_2_517x315.png" alt="image" data-base62-sha1="8YidKYE3191cxernNKVeDzwocEu" width="517" height="315" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3ee231b3cbd63d5af8603d624e59a2d0238a2a9e_2_517x315.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3ee231b3cbd63d5af8603d624e59a2d0238a2a9e_2_775x472.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3ee231b3cbd63d5af8603d624e59a2d0238a2a9e_2_1034x630.png 2x" data-dominant-color="111F24"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1430×874 48.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And actually, I have some other containers that are not running any scripts now.<br>
I thought each docker container can fully utilize the GPU resource when the GPU-Util is 0%, but at the same time I find in the last row it says that about 36GB of GPU is already in-use.</p>
<p>Does this probably mean even if the other containers are not using GPU to run scripts currently, each container automatically limits its maximum GPU resource reserved if there’re multiple containers?</p>
<p>Thank you.</p>

ptrblck: <p>No, docker containers are not limiting the GPU resources (there might be options to do so, but I’m unaware of these).<br>
As you can see in the output of <code>nvidia-smi</code> 4 processes are using the device where the Python scripts are taking the majority of the GPU memory so the OOM error would be expected.</p>
<aside class="quote no-group" data-username="HitM" data-post="6" data-topic="149037">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/cc9497/48.png" class="avatar"> HitM:</div>
<blockquote>
<p>I thought each docker container can fully utilize the GPU resource when the GPU-Util is 0%, but at the same time I find in the last row it says that about 36GB of GPU is already in-use.</p>
</blockquote>
</aside>
<p>The “GPU-Util.” shows the percentage of the kernel execution time in the last time frame, i.e. it’s showing the compute resource usage not the memory usage.</p>

HitM: <p>Hi ptrblck,</p>
<p>Thank you for your insight.<br>
Then I was misunderstanding the output of nvidia-smi.<br>
I will find and kill the processes that are using huge resources and confirm if PyTorch can reserve larger GPU memory.</p>
<p>→I confirmed that both of the processes using the large resources are in the same docker container. As I was no longer running scripts in that container, I feel it was strange. Anyway, I killed the processes and I’m running my code again to check if I still encounter the issue or not.</p>
<p>Thank you.</p>

HitM: <p>It seems to be working w/o causing the OOM error. I’m not sure why the container no longer running scripts was occupying huge GPU memory, but once I killed these processes, I no longer encountered the problem.<br>
Thank you both for providing me with the insights!</p>

HitM: <aside class="quote no-group" data-username="HitM" data-post="9" data-topic="149037">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/cc9497/48.png" class="avatar"> HitM:</div>
<blockquote>
<p>I’m not sure why the container no longer running scripts was occupying huge GPU memory</p>
</blockquote>
</aside>
<p>FYI: I have confirmed that the process that had been terminated with Ctrl+Z was occupying memory. I sometimes use the command when Ctrl+C doesn’t work and this was the reason. This was solved by deleting the Docker container.</p>

redradist: <p>I have the similar issue:</p>
<pre><code class="lang-auto">torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.15 GiB. GPU 0 has a total capacty of 11.72 GiB of which 826.50 MiB is free. Including non-PyTorch memory, this process has 10.91 GiB memory in use. Of the allocated memory 7.67 GiB is allocated by PyTorch, and 3.03 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
&lt;user&gt;@&lt;machine&gt;whisper-finetuning(main)$ nvidia-smi 
Wed Nov 22 00:24:40 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4070        On  | 00000000:01:00.0 Off |                  N/A |
| 30%   36C    P8              10W / 200W |      2MiB / 12282MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
</code></pre>
<p>Looks like before running the process I have a lot of GPU memory, but PyTorch reserve 3.03 GiB but do not allocate it …</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> Do you know how to fix it ? Could you help with it ?</p>

Manoj_Prasad: <p>i am facing the same error if u have solved it can u please provide me the solution ?</p>