Memory leaks in Pytorch object detection

username: tengerye
cooked: <p>I am working on the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#putting-everything-together" rel="nofollow noopener">object detection tutorial</a> on PyTorch. The original tutorial works fine with the few epochs given. I expanded it to large epochs and encounter <strong>out of memory</strong> error.</p>
<p>I tried to debug it and find something interesting. This is the tool I am using:</p>
<pre><code class="lang-auto">def debug_gpu():
    # Debug out of memory bugs.
    tensor_list = []
    for obj in gc.get_objects():
        try:
            if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
                tensor_list.append(obj)
        except:
            pass
    print(f'Count of tensors = {len(tensor_list)}.')
</code></pre>
<p>And I used it to monitor the memory of training one epoch:</p>
<pre><code class="lang-auto">def train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq):
    ...
    for images, targets in metric_logger.log_every(data_loader, print_freq, header):
        # inference + backward + optimization
        debug_gpu()
</code></pre>
<p>The output is something like this:</p>
<pre><code class="lang-auto">Count of tensors = 414.
Count of tensors = 419.
Count of tensors = 424.
Count of tensors = 429.
Count of tensors = 434.
Count of tensors = 439.
Count of tensors = 439.
Count of tensors = 444.
Count of tensors = 449.
Count of tensors = 449.
Count of tensors = 454.
</code></pre>
<p>As you can see, the count of tensors tracked by garbage collector increases constantly.</p>
<p>Relevant files to execute can be found <a href="https://github.com/pytorch/vision/tree/master/references/detection" rel="nofollow noopener">here</a>.</p>
<p>I have two questions:</p>
<ol>
<li>What is holding up the garbage collector to release these tensors?</li>
<li>What should I do with the out of memory error?</li>
</ol>

username: fmassa
cooked: <p>Hi,</p>
<p>I think we might need some more information before we can understand where the problem is.</p>
<ul>
<li>are you using multiple GPUs to train the model?</li>
<li>can you remove the <code>metric_logger</code> logging, and just iterate over the dataloader</li>
</ul>
<p>Those are the two things that came to my mind. I haven’t experienced this OOM before, so I’m not sure what else it could be for now</p>

username: tengerye
cooked: <p>Hi,<br>
Thank you for the kind reply.</p>
<ol>
<li>I used one gpu. But even if I used CPU, the situation happens;</li>
<li>I did try to remove <code>metric_logger</code>, but it still happens. I doubt if the model itself caused it.</li>
</ol>
<p>The original tutorial has such few epochs to trigger the OOM, but I believe you can easily repeat my situation with my code.</p>

username: tengerye
cooked: <p>Hi <a class="mention" href="/u/fmassa">@fmassa</a>, I think I found the reason: <code>rpn.anchor_generator._cache</code> keeps the pair with the model and the number of tensors it holds expands as proposal grows. I provide a <a href="https://github.com/tengerye/display_memory_leak" rel="nofollow noopener">sample</a> for an easy repeat.</p>
<p>BTW, I send a pull <a href="https://github.com/pytorch/vision/pull/1657" rel="nofollow noopener">request</a> with the fix along with other data type error solution but codecov failed me. Would you please have a look?</p>

username: albanD
cooked: <p>PR was merged. Thanks for the contribution <a class="mention" href="/u/tengerye">@tengerye</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>