Memory usage increasing with number of steps while training

Yash_Rathi: <p>Hello<br>
As I train the memory usage keeps increasing very rapidly as the number of steps increase.<br>
I have also tried deleting all variables after every iteration.<br>
Train loop:</p>
<pre data-code-wrap="py"><code class="lang-py">def train_epoch(model: TransformerNetwork, optimizer):
    global mem_used_arr_mbs
    mem_used_arr_mbs = []
    model.train()
    losses = 0
    train_iter = EnHindiDataset(split='train', language_pair=(SRC_LANGUAGE, TGT_LANGUAGE), limit=200_000)
    train_dataloader = DataLoader(train_iter, batch_size=BATCH_SIZE, collate_fn=collate_fn)

    for src, tgt in tqdm(train_dataloader):
        mem_used = int(get_gpu_memory_usage().split()[0])
        mem_used_arr_mbs.append(mem_used)
        src = src.to(DEVICE)
        tgt = tgt.to(DEVICE)

        tgt_input = tgt[:-1, :]

        src_mask, tgt_mask, src_padding_mask, tgt_padding_mask = model.create_mask(src, tgt_input)

        logits = model(src, tgt_input, src_mask, tgt_mask,src_padding_mask, tgt_padding_mask, src_padding_mask)

        optimizer.zero_grad()

        tgt_out = tgt[1:, :]
        loss = loss_fn(logits.reshape(-1, logits.shape[-1]), tgt_out.reshape(-1))
        loss.backward()

        optimizer.step()
        losses += loss.item()

        del src, tgt, tgt_input
        del src_mask, tgt_mask, src_padding_mask, tgt_padding_mask
        del logits, tgt_out, loss


    return losses / len(train_dataloader)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/3/03b22052331424467f0c24426785d0db66dec32e.png" alt="image" data-base62-sha1="wH46hDUzoZEAEnGStQo0DlSvJc" width="476" height="278"></p>
<p>Graph of memory usage vs n_steps. On x-axis are the steps and on y is the memory usage in mbs.<br>
I don’t understand why the memory usage increases after each step, as pytorch don’t even need to store any information about the last step.</p>
<p>Memory consumption with time:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9fd8e420820a016c6b0d07355808fc647e1cfe48.png" alt="image" data-base62-sha1="mO4Bvb3qYoizbdt8RPtK1grG0nm" width="620" height="331"></p>
<p>After the epoch is some % complete I get this error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/058cc0a3ffd5db412b72a18d42de2bcb4405044d.png" data-download-href="https://discuss.pytorch.org/uploads/default/058cc0a3ffd5db412b72a18d42de2bcb4405044d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/058cc0a3ffd5db412b72a18d42de2bcb4405044d_2_690x32.png" alt="image" data-base62-sha1="N5WMS74TO1qUPIQTs00EI62Z6l" width="690" height="32" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/058cc0a3ffd5db412b72a18d42de2bcb4405044d_2_690x32.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/058cc0a3ffd5db412b72a18d42de2bcb4405044d_2_1035x48.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/5/058cc0a3ffd5db412b72a18d42de2bcb4405044d.png 2x" data-dominant-color="474747"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1247×59 19.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please help<br>
Kindly inform me if any more information is needed.</p>

Yash_Rathi: <p>Here’s the full colab notebook if helps <a href="https://colab.research.google.com/drive/15sA0ArIv5QFlp8Q1V_htrVr0OcqrA71w?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>

eqy: <p>In this situation one way to isolate the source of the growing memory usage is to remove parts of the training loop incrementally to see when the memory usage stops increasing.</p>

Yash_Rathi: <p>yes but since the training loop is already very simple, I don’t think I could remove anything else from it. I also tried on a 16gb gpu and it was able to complete 1 epoch, and the memory usage is stable at 97% right now. Though it might crash later</p>

di0002ya: <p>I encounter similar problem. What’s the version of your pytorch? Have u solved it?</p>