Not releasing GPU memory - how to debug?

username: motown_dad
cooked: <p>I have a flask (python) server, serving pytorch models as API.<br>
With each request allocated GPU memory grows and eventually I get “Out of Memory”.<br>
As far as I can tell, there shouldn’t be anything in my code holding any references to models/tensors, and everything should be initialized from scratch for each request, but I’m sure I’m wrong, just don’t know how to debug: what is holding reference to these tensors?</p>
<p>I filtered out the tensors on GPU:</p>
<pre><code class="lang-auto">    import gc
    objs = gc.get_objects()
    gpu_tensors = [obj for obj in objs if isinstance(obj, torch.Tensor) and obj.is_cuda]
</code></pre>
<p>I can get objects referring to these tensors using <code>gc.get_referrers(t)</code> – most of them are referenced by three: “&lt;class ‘list’&gt;”, “&lt;class ‘list’&gt;”, “&lt;class ‘collections.OrderedDict’&gt;” (and few by 72 – probably suspicious?)<br>
<strong>But I don’t know how to figure out what is this “list” in my code? Why variable is holding it so I can delete/release it?</strong></p>
<p>I also tried to visualize using <code>objgraph</code>, and get nice graphs, like this one for one of those tensors:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e39b56b90ea90ee6a8e504e9eb326b79e30a2d4f.png" data-download-href="https://discuss.pytorch.org/uploads/default/e39b56b90ea90ee6a8e504e9eb326b79e30a2d4f" title="backrefs_140020672845664"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e39b56b90ea90ee6a8e504e9eb326b79e30a2d4f_2_690x411.png" alt="backrefs_140020672845664" data-base62-sha1="wtvcfopwA4vu2fLMJrFr0nqKbKL" width="690" height="411" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e39b56b90ea90ee6a8e504e9eb326b79e30a2d4f_2_690x411.png, https://discuss.pytorch.org/uploads/default/original/3X/e/3/e39b56b90ea90ee6a8e504e9eb326b79e30a2d4f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/3/e39b56b90ea90ee6a8e504e9eb326b79e30a2d4f.png 2x" data-dominant-color="E7E6E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">backrefs_140020672845664</span><span class="informations">900×537 37.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<strong>But the same problem – where in my code is this “list” or “list_iterator”?</strong></p>
<p>I would be grateful for any tips on how to learn to debug this issue.</p>

username: Aniruth_Sundararaja1
cooked: <p>try running torch.cuda.empty_cache() at the beginning.</p>
<p>This cleares all cached memory in gpu</p>

username: motown_dad
cooked: <p>Thank you. I already did that, but it didn’t change anything.</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/motown_dad">@motown_dad</a>  If possible can you share your flask code?</p>

username: ptrblck
cooked: <p>Clearing the cache won’t help as described in other posts, since only the reusable memory will be returned to the system. It will however slow down your code as the <code>cudaFree</code> calls are synchronizing.</p>

username: motown_dad
cooked: <p>Here is the code: <a href="https://gist.github.com/motowndad/63839a4990fb8d1dbc383d3c2d0fea09" class="inline-onebox" rel="noopener nofollow ugc">ComfyUI workflow for debugging · GitHub</a></p>
<p>It’s generated from ComfyUI workflow, using (ComfyUI-to-Python-Extension)[<a href="https://github.com/pydn/ComfyUI-to-Python-Extension" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pydn/ComfyUI-to-Python-Extension: A powerful tool that translates ComfyUI workflows into executable Python code.</a>]</p>
<p>I don’t expect anyone to debug layers of frameworks of my code (in this case bunch of ComfyUI classes and then pytorch underneath). I’m hoping for advice how I can debug this myself.</p>

username: motown_dad
cooked: <p>That makes sense. Any advice how to debug this? How can I figure out what are these referrers that <code>objgraph</code> or <code>gc</code> is showing?</p>