The running training is being killed when a new one is scheduled on A6000 GPU

edikmkoyan: <p>I have 250+ GB of RAM and 48 GB on A6000<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/8/080715a0d0437f4be7848e065e1a4932bb8e08aa.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/080715a0d0437f4be7848e065e1a4932bb8e08aa" title="Screenshot from 2023-05-12 15-02-42"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/080715a0d0437f4be7848e065e1a4932bb8e08aa_2_690x226.jpeg" alt="Screenshot from 2023-05-12 15-02-42" data-base62-sha1="190ZT5GJOAqm1bu5Gj35kTQ6lf4" width="690" height="226" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/080715a0d0437f4be7848e065e1a4932bb8e08aa_2_690x226.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/080715a0d0437f4be7848e065e1a4932bb8e08aa_2_1035x339.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/080715a0d0437f4be7848e065e1a4932bb8e08aa_2_1380x452.jpeg 2x" data-dominant-color="3F1933"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-05-12 15-02-42</span><span class="informations">1600×525 214 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You are running out of GPU memory and would need to reduce the memory usage in each process. You could use <code>torch.cuda.set_per_process_memory_fraction</code> to limit the fraction per process.</p>

edikmkoyan: <p>Thanks. I also tried to follow this guide, but didn’t find working way to set the memory allocation strategy, can you please assist? How can I set it to heuristic?<br>
<a href="https://iamholumeedey007.medium.com/memory-management-using-pytorch-cuda-alloc-conf-dabe7adec130" class="onebox" target="_blank" rel="noopener nofollow ugc">https://iamholumeedey007.medium.com/memory-management-using-pytorch-cuda-alloc-conf-dabe7adec130</a></p>

edikmkoyan: <p>One more thing. Suppose I have a training that may potentially use all the 48 GB of the GPU memory, in such case I will set the <code>torch.cuda.set_per_process_memory_fraction</code> to 1. It starts running knowing that it can allocate all the memory, but it didn’t yet. Another process starts and tries to allocate some significant chunk of memory, which process will be killed?<br>
I also didn’t mentioned that oom killer kills the process although the process does not consume system RAM, it consumes GPU RAM, can oom killer do it’s scoring based on GPU RAM?</p>
<p>Here is the dmesg log for killed trainings.</p>
<pre><code class="lang-auto">[1049496.935479] **Out of memory**: Killed process 3718080 (python) total-vm:40401236kB, anon-rss:6162012kB, file-rss:68616kB, shmem-rss:287344kB, UID:1001 pgtables:18492kB oom_score_adj:0

[1049506.723130] **Out of memory**: Killed process 3717888 (python) total-vm:40402784kB, anon-rss:6163596kB, file-rss:68064kB, shmem-rss:313964kB, UID:1001 pgtables:18548kB oom_score_adj:0

[1049519.565244] **Out of memory**: Killed process 3717292 (python) total-vm:40445528kB, anon-rss:6206448kB, file-rss:68480kB, shmem-rss:366624kB, UID:1001 pgtables:18736kB oom_score_adj:0

[1049535.486537] **Out of memory**: Killed process 3718466 (python) total-vm:40458440kB, anon-rss:6219144kB, file-rss:68788kB, shmem-rss:343796kB, UID:1001 pgtables:18716kB oom_score_adj:0
</code></pre>

ptrblck: <aside class="quote no-group" data-username="edikmkoyan" data-post="4" data-topic="186752">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/edikmkoyan/48/63337_2.png" class="avatar"> edikmkoyan:</div>
<blockquote>
<p>which process will be killed</p>
</blockquote>
</aside>
<p>Unclear, but I would assume the process trying to allocate memory and failing will raise the error.</p>
<aside class="quote no-group" data-username="edikmkoyan" data-post="4" data-topic="186752">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/edikmkoyan/48/63337_2.png" class="avatar"> edikmkoyan:</div>
<blockquote>
<p>that oom killer kills the process although the process does not consume system RAM</p>
</blockquote>
</aside>
<p>I doubt it as the OS doesn’t care about GPU memory but will kill the process if it’s running out of host RAM.</p>

edikmkoyan: <p>That is strange, as the python processes that had been killed by oom, were not using System RAM, in fact as we may see in the image attached, the GPU memory is consumed. In fact We’ve never used even 50% of the System RAM on the machine.</p>
<p>Can pytorch set the OOM score somehow to influence the OOM Killer behaviour ?</p>

ptrblck: <aside class="quote no-group" data-username="edikmkoyan" data-post="6" data-topic="186752">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/edikmkoyan/48/63337_2.png" class="avatar"> edikmkoyan:</div>
<blockquote>
<p>That is strange, as the python processes that had been killed by oom, were not using System RAM,</p>
</blockquote>
</aside>
<p>This is impossible as Python will of course use system RAM to store its runtime, variables etc.<br>
Also the OOM message shows the used memory:</p>
<pre><code class="lang-auto">[1049535.486537] **Out of memory**: Killed process 3718466 (python) total-vm:40458440kB, anon-rss:6219144kB, file-rss:68788kB, shmem-rss:343796kB, UID:1001 pgtables:18716kB oom_score_adj:0
</code></pre>
<p>as virtual memory and resident memory.</p>

edikmkoyan: <p>Sorry my bad, I mean it wasn’t using that much memory, we hardly use 50% of our System RAM, System RAM congestion could not be the reason for OOM Killer  wake up.</p>