[Solved] Memory Usage Increasing Constantly

Mandias: <p>While training an autoencoder my memory usage will constantly increase over time (using up the full ~64GB available). Initially I thought it was just the loss function, buy I get the same behavior with both BCELoss and MSELoss.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b8e9dcd6783ba1e9243ee67d6e491e8ad86fa38.png" alt="image" data-base62-sha1="3VMrEhubAYlTAJc0rCWyS8oxw36" width="675" height="373"></p>
<p>I’ve read the FAQ about memory increasing and ensured that I’m not unintentionally keeping gradients in memory.</p>
<p>Attempting to split the data into mini-batches (“chunks” in the code example) does not change the behavior at all.</p>
<pre><code class="lang-auto">    opt = torch.optim.Adam(net.parameters())
    loss_fn = nn.BCELoss()
    # for reducing RAM usage use subsets of training data
    subsets = 8
    chunks = train_ingr.split(len(indexes)//subsets)
    for i in range(epochs):
        loss = loss_fn(net(chunks[i%subsets]), chunks[i%subsets])
        loss.backward()
        train_loss_hist.append(loss.detach().numpy())
        opt.step()
        opt.zero_grad()
</code></pre>
<p>I don’t think I’m doing anything in particular wrong. The network being trained is a simple dense linear autoencoder with activations. Everything is on CPU, so there shouldn’t be accidental copies created by moving data around.</p>
<p>Please let me know if I’ve missed something important.</p>
<p>Edit: adding memory profiling results:</p>
<pre><code class="lang-auto">-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem    # of Calls
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                                    aten::empty_strided         0.10%      41.003ms         0.10%      41.003ms       1.544us      48.96 Gb      48.96 Gb         26562
                                            aten::addmm         6.94%        2.900s        10.67%        4.457s       1.431ms      26.50 Gb      26.50 Gb          3114
                                         aten::_softmax        11.82%        4.940s        11.82%        4.940s       9.518ms      25.85 Gb      25.85 Gb           519
                           aten::_softmax_backward_data         9.73%        4.064s         9.73%        4.064s       8.127ms      24.02 Gb      24.02 Gb           500
                                       aten::empty_like         0.02%       8.228ms         0.08%      35.009ms      22.689us      49.73 Gb       2.55 Gb          1543
                                               aten::mm        13.40%        5.600s        13.40%        5.600s       1.018ms     821.60 Mb     821.58 Mb          5500
                                             aten::mish         3.77%        1.574s         3.77%        1.574s     758.274us     652.97 Mb     652.97 Mb          2076
                                    aten::mish_backward         4.81%        2.009s         4.81%        2.012s       1.006ms     605.75 Mb     605.75 Mb          2000
                                           aten::linear         0.12%      52.147ms        10.79%        4.509s       1.448ms      26.50 Gb     289.33 Mb          3114
                                             aten::sqrt         0.10%      41.426ms         0.10%      41.426ms       6.904us     205.93 Mb     205.93 Mb          6000
                                              aten::div         0.11%      44.089ms         0.20%      84.888ms      14.148us     205.93 Mb     205.46 Mb          6000
                                             aten::tanh         0.03%      10.577ms         0.03%      10.577ms      20.380us      16.32 Mb      16.32 Mb           519
                                    aten::tanh_backward         0.02%       7.644ms         0.02%       7.644ms      15.288us      14.93 Mb      14.93 Mb           500
                                          MishBackward0         0.23%      94.971ms         4.83%        2.018s       1.009ms     590.21 Mb      12.26 Mb          2000
                                              aten::sum         5.38%        2.246s         5.42%        2.264s     643.471us       6.34 Mb       6.31 Mb          3519
                                               aten::to         0.10%      41.906ms         0.39%     163.319ms       6.392us     529.35 Kb     441.54 Kb         25550
                                            aten::zero_         0.00%      55.000us         0.00%     198.000us       8.250us     202.69 Kb     202.69 Kb            24
                                         aten::_to_copy         0.25%     102.501ms         0.32%     133.250ms       5.326us     112.85 Kb      37.05 Kb         25019
                                            aten::empty         0.01%       4.462ms         0.01%       4.462ms       1.763us      31.47 Kb      31.47 Kb          2531
                                     aten::resolve_conj         0.00%     231.000us         0.00%     231.000us       0.011us      15.75 Kb      15.75 Kb         20247
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
</code></pre>

Mandias: <p>Tried again with different loss functions and optimizers.</p>
<p><strong>Optimizers</strong><br>
SGD: No difference, memory increases in the same way<br>
Adadelta: No difference, memory increases in the same way</p>
<p><strong>Loss Functions</strong><br>
A custom implementation of Dice Loss: no problem, there’s barely any increase in memory over time:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/053e109b02e995c2e0d9d6e868413e6b394aa5b6.png" data-download-href="https://discuss.pytorch.org/uploads/default/053e109b02e995c2e0d9d6e868413e6b394aa5b6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/053e109b02e995c2e0d9d6e868413e6b394aa5b6.png" alt="image" data-base62-sha1="KnmmzYkeZBefIZ4o6m4ZV9imCa" width="690" height="355" data-dominant-color="EAEEF7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">719×370 15.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It looks like the other loss functions (BCELoss and MSELoss) are related to the memory issue. Unfortunately, BCELoss is the only one that works well for my application so far.</p>
<p>In case there was something being stored internal to the loss function I tried creating a new instance of  it (nn.BCELoss) each epoch, but this had no effect on the memory usage (still constantly increasing).</p>
<p>I’m guessing that the issue is a combination of data being generated by the loss function and stored by the optimizer. Does that sound right? Is there a good way to address the issue?</p>
<p>(BTW, I’ve tried using half-precision but there’s a know bug that makes it tremendously slow on CPU)</p>

Mandias: <p>On a whim I changed from <code>loss.detach().numpy()</code> to <code>loss.item()</code> and the memory issues <strong>disappeared</strong>.<br>
Could someone explain if this makes sense? I thought they would be roughly equivalent for my setup.</p>
<p>What’s even more strange is that this behavior was not consistent across loss functions (a bug?).</p>