Out of Memory and Can't Release GPU Memory

username: NAN_JIANG
cooked: <p>I use try-catch to enclose the forward and backward functions, and I also delete all the tensors after every batch.</p>
<pre><code class="lang-auto">            try:
                decoder_output, loss = model(batch)
                if Q.qsize() &gt; 0:
                    break
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
                train_loss.append(loss.mean().item())
                del decoder_output, loss
            except Exception as e:
                optimizer.zero_grad()
                for p in model.parameters():
                    if p.grad is not None:
                        del p.grad
                torch.cuda.empty_cache()
                oom += 1
            del batch
            for p in model.parameters():
                if p.grad is not None:
                    del p.grad
            torch.cuda.empty_cache()
</code></pre>
<p>The training process is normal at the first thousands of steps, even if it got OOM exception, the exception will be catched and the GPU memory will be released.</p>
<p>But after I trained thousands of batches, it suddenly keeps getting OOM for every batch and the memory seems never be released anymore.</p>
<p>It’s so weird to me, is there any suggestions? (I’m using distributed data parallel)</p>

username: NAN_JIANG
cooked: <p>Here is the code before the try-catch, i.e., the code to prepare input data with train_loader:</p>
<pre><code class="lang-auto">        for i, batch in enumerate(train_loader):
            batch = {k: v.squeeze(0).cuda(non_blocking=True) for (k, v) in batch.items()}
            batch['rem_father_index'] = torch.split(batch['rem_father_index'], batch['rem_root_num'].tolist(), dim=0)
            batch['rem_father_index'] = [l.tolist() for l in batch['rem_father_index']]
            batch['tree_sizes'] = batch['tree_sizes'].tolist()
</code></pre>

username: NAN_JIANG
cooked: <p>I have some updates:</p>
<ol>
<li>If I create some tensors like: torch.zeros(bsz, max_len), will this cause some side-effect?</li>
<li>I output the python garbage collector, and find something wierd:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d1d2dec72259e3c95be1208c129efe26cbd5e82d.png" data-download-href="https://discuss.pytorch.org/uploads/default/d1d2dec72259e3c95be1208c129efe26cbd5e82d" title="Screenshot from 2021-09-23 12-53-45"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d1d2dec72259e3c95be1208c129efe26cbd5e82d.png" alt="Screenshot from 2021-09-23 12-53-45" data-base62-sha1="tWbzSwiMv0YmzeSQ8fmgZ5G5FPn" width="690" height="315" data-dominant-color="363731"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-09-23 12-53-45</span><span class="informations">816×373 46.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
There is a huge dict, which contains a lot of tensors, is this part of the graph or something?</li>
</ol>

username: mutiann
cooked: <p>I found something similar now…Did you find any solution?</p>

username: NAN_JIANG
cooked: <p>This issue only occurs with DistributedDataParallel.</p>
<p>Finally, I go back to DataParallel and it doesn’t have the problem, although it’s slower.</p>

username: mutiann
cooked: <p>It happens to me without DDP <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"><br>
I opened an issue here, possibly related: <a href="https://github.com/pytorch/pytorch/issues/82218" class="inline-onebox" rel="noopener nofollow ugc">OOM during backward() leads to memory leaks · Issue #82218 · pytorch/pytorch · GitHub</a></p>