Systematically debugging out-of-memory issue

Roee_Shenberg: <p>I’m having a recurring out-of-memory issue that seems to be caused by memory fragmentation:</p>
<pre><code class="lang-auto">torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 23.65 GiB total capacity; 12.35 GiB already allocated; 2.56 MiB free; 22.33 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
</code></pre>
<p>Note: allocation attempt is 2MB, card has 24GB of RAM, 12.35GB already allocated, only 2.56MB free out of 22GB reserved.</p>
<p>Things I considered already:</p>
<ol>
<li>This is training of an RNN on variable sequence lengths. I set <code>drop_last=True</code> in the dataloader, and I tried forcing the first batch to have the longest sequence, which does not cause an instantaneous crash. It crashes only after some epochs, so it’s not a momentary issue caused by the unusual size of a batch.</li>
<li>I checked to see that all tensors retained for logging are either detached or have <code>.item()</code> called on them.</li>
<li>It’s not a cuda cache filling up issue: Every training epoch ends with a call to <code>torch.cuda.empty_cache()</code></li>
<li>It’s not a hardware issue: the issue recurred on separate machines, and the main machine has two GPUs and behavior was identical independently from which GPU the code was using.</li>
<li>Memory leaks: Maybe I don’t understand something but I have evidence to the contrary.</li>
</ol>
<p>My next step was to add logging of <code>torch.cuda.memory_stats()</code> to tensorboard as part of training to see if memory is leaking. I log every 500 steps (roughly 1.5 epochs), and I would expect to see <code>allocation.*.current</code> and <code>allocated_bytes.*.current</code> to increase (where * is one of large, small, all). Instead what I see is that all these metrics are roughly static - there’s noise but no obvious trend across the roughly 9000 steps until the crash.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/86f0458969ad65f965b1369ebb96e82ffbeb4da4.png" data-download-href="https://discuss.pytorch.org/uploads/default/86f0458969ad65f965b1369ebb96e82ffbeb4da4" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/6/86f0458969ad65f965b1369ebb96e82ffbeb4da4.png" alt="image" data-base62-sha1="jfIL9Rc8G1FGMcrOA111QRl2fhG" width="621" height="500" data-dominant-color="F7F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">624×502 33.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fcabefbfc9aace52d30293e38ddf6a5dab957bfa.png" data-download-href="https://discuss.pytorch.org/uploads/default/fcabefbfc9aace52d30293e38ddf6a5dab957bfa" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fcabefbfc9aace52d30293e38ddf6a5dab957bfa.png" alt="image" data-base62-sha1="A3eH4ZUW8EqQhztQQ5LDT6cum5s" width="601" height="500" data-dominant-color="F7F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">614×510 35.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1b9bed5be994d3dc2db6a07dd17193cdbe0838e.png" data-download-href="https://discuss.pytorch.org/uploads/default/f1b9bed5be994d3dc2db6a07dd17193cdbe0838e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1b9bed5be994d3dc2db6a07dd17193cdbe0838e.png" alt="image" data-base62-sha1="yup1Kh3zije3bR03HtsrSytOWjc" width="606" height="500" data-dominant-color="F8F5F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">616×508 33.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Seems like a slam-dunk for a memory fragmentation issue and not a leak, right? But, I would expect one of <code>inactive_split_bytes.*.current</code> and/or <code>inactive_splits.*.current</code> to display an increasing trend, but that’s not what I see.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf9eab381695e52d08746ed927470bed8c192eba.png" alt="image" data-base62-sha1="tCGMwYrp1GzdzcvqyRkkDNAOftE" width="628" height="490"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2d97c763b2d965ce200b0dc4565c93558280973.png" data-download-href="https://discuss.pytorch.org/uploads/default/d2d97c763b2d965ce200b0dc4565c93558280973" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2d97c763b2d965ce200b0dc4565c93558280973.png" alt="image" data-base62-sha1="u5gej8henV18bnFuFn7OHEbsoLx" width="596" height="500" data-dominant-color="F7F3F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">618×518 35.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6466b03ce066f864e40dbecdcad7e1eb3fb0957.png" alt="image" data-base62-sha1="si1pmu9Dvh3aOIWDcwup2u3SOfJ" width="620" height="494"></p>
<p>Having said all that, the exception shows that free memory is much less than (reserved memory - allocated memory), so I’m wrong somewhere down the line - it’s very probably a memory fragmentation issue.</p>
<p>The way I see it is that I misunderstood how the metrics are supposed to show memory issues. Any ideas on metrics that will reliably show memory fragmentation?</p>
<p>Also, another issue I don’t understand really is setting <code>max_split_size_mb</code> - any recommendations there? I mean, I don’t understand my model’s memory allocation behavior, I suspect I need to in order to set the value intelligently. Any recommendations on how to gather this information and how to choose the value?</p>
<p>Thanks!</p>

ptrblck: <p>Thanks for the detailed summary!</p>
<aside class="quote no-group" data-username="Roee_Shenberg" data-post="1" data-topic="175034">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/roee_shenberg/48/12891_2.png" class="avatar"> Roee_Shenberg:</div>
<blockquote>
<p>Having said all that, the exception shows that free memory is much less than (reserved memory - allocated memory), so I’m wrong somewhere down the line - it’s very probably a memory fragmentation issue.</p>
</blockquote>
</aside>
<p>Could you check the reported memory usage in <code>nvidia-smi</code> additionally during the training to see if you might see a real memory leak? The memory summary reported by PyTorch will not check the memory usage of other processes and would not be able to report leaked memory (otherwise it wouldn’t be a leak).</p>

Roee_Shenberg: <p>When I gave it an anecdotal looks once every few hours, <code>nvidia-smi</code> aligned pretty well with reserved memory as reported by pytorch, and python is the only process on the GPU except for <code>/usr/lib/xorg/Xorg</code> which is using 4MB.</p>
<p>However, I found a small issue with the training code which I think was the cause:<br>
The <code>train_epoch()</code> function looks something like this</p>
<pre><code class="lang-python">for batch in data_loader:
    outputs = train_step(model, batch) # does forward, backward and optimize
    if outputs is None:
        # warn
        continue
    log_outputs(outputs)
torch.cuda.empty_cache()
</code></pre>
<p>The problem is that technically you can only free the previous loop iteration’s <code>outputs</code> after running <code>train_step()</code>, since that is when the reference is updated. I added an explicit <code>del outputs</code> in the end of the loop body and training has stopped crashing. Unfortunately, this was not a systematic solution, rather an educated guess, but at least it works.</p>