CPU RAM Exploded

username: wwaayyaaww
cooked: <p>Hi. I failed to trace the reason why the CPU RAM usage increases after every iteration and exploded after some hundred of iterations.</p>
<p>I have replaced the <code>loss += loss_batch</code> with <code>loss += loss_batch.detach()</code>.</p>
<p>Unfortunately, the CPU RAM problem remains the same.</p>
<p>My codes are as follows:</p>
<pre><code class="lang-auto">    for batch_idx in range (iter_max):

        lfw_lr_in, lfw_lr_in_aug_1, lfw_lr_in_aug_2 = next(lfw_loader)
        
        lfw_lr_x, lfw_lr_y = lfw_lr_in
        lfw_lr_x_aug_1, lfw_lr_y_aug_1 = lfw_lr_in_aug_1
        lfw_lr_x_aug_2, lfw_lr_y_aug_2 = lfw_lr_in_aug_2
             
        assert torch.all(torch.eq(lfw_lr_y, lfw_lr_y_aug_1))
        assert torch.all(torch.eq(lfw_lr_y, lfw_lr_y_aug_2))
        del lfw_lr_in, lfw_lr_in_aug_1, lfw_lr_in_aug_2
        
        # ***
        
        lfw_lr_x = lfw_lr_x.to(device)
        lfw_lr_y = lfw_lr_y.to(device)

        lfw_lr_emb = model(lfw_lr_x)

        del lfw_lr_x
        
        # ***

        lfw_lr_x_aug_1 = lfw_lr_x_aug_1.to(device)
        lfw_lr_y_aug_1 = lfw_lr_y_aug_1.to(device)

        lfw_lr_emb_aug_1 = model(lfw_lr_x_aug_1)

        del lfw_lr_x_aug_1 

       # ***

        lfw_lr_x_aug_2 = lfw_lr_x_aug_2.to(device)
        lfw_lr_y_aug_2 = lfw_lr_y_aug_2.to(device)

        lfw_lr_emb_aug_2 = model(lfw_lr_x_aug_2)

        del lfw_lr_x_aug_2
                
        # ***
        
        lfw_lr_emb = torch.cat((lfw_lr_emb, lfw_lr_emb_aug_1, lfw_lr_emb_aug_2), dim = 0)
        lfw_lr_y = torch.cat((lfw_lr_y, lfw_lr_y_aug_1, lfw_lr_y_aug_2))
        
        del lfw_lr_emb_aug_1, lfw_lr_y_aug_1
        del lfw_lr_emb_aug_2, lfw_lr_y_aug_2
                
        # *** *** 

        # Estimate lfw_loss_ce
        lfw_loss_ce = 0
        
        if net_params['lfw_fc_ce_flag'] is True:

            lfw_lr_pred = lfw_fc(lfw_lr_emb, lfw_lr_y)
            lfw_loss_ce = loss_fn['loss_ce'](lfw_lr_pred, lfw_lr_y)
                            
        del lfw_lr_emb, tf_lr_emb
        
        # *** ***
                        
        # Define loss_batch 
        loss_batch = lfw_loss_ce
                
        # *** ***

        # if model.training:
        optimizer.zero_grad()
        loss_batch.backward() 
        optimizer.step()
        
        time.sleep(0.0001)
        
        # *** ***
        
        metrics_batch = {}
        for metric_name, metric_fn in batch_metrics.items():
            metrics_batch[metric_name] = metric_fn(tf_lr_pred, tf_lr_y).detach().cpu()
            metrics[metric_name] = metrics.get(metric_name, 0) + metrics_batch[metric_name]
            
        if writer is not None: # and model.training:
            if writer.iteration % writer.interval == 0:
                writer.add_scalars('loss', {mode: loss_batch.detach().cpu()}, writer.iteration)
                for metric_name, metric_batch in metrics_batch.items():
                    writer.add_scalars(metric_name, {mode: metric_batch}, writer.iteration)
            writer.iteration += 1
        
        #
        #
        # loss += loss_batch
        loss += loss_batch.detach()
        if show_running:
            logger(loss, metrics, batch_idx)
        else:
            logger(loss_batch, metrics_batch, batch_idx)
            
        gc.collect()
            
    # END FOR
 
    # *** ***
        
    # if model.training and scheduler is not None:
    if scheduler is not None:
        scheduler.step()

    loss = loss / (batch_idx + 1)
    metrics = {k: v / (batch_idx + 1) for k, v in metrics.items()}
    
    return metrics
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30a219f872ff98669e9092ce1b51a2fc97c73f31.png" data-download-href="https://discuss.pytorch.org/uploads/default/30a219f872ff98669e9092ce1b51a2fc97c73f31" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30a219f872ff98669e9092ce1b51a2fc97c73f31_2_690x374.png" alt="image" data-base62-sha1="6WedB6D2nyGO3qHleAB7vTDZgBj" width="690" height="374" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30a219f872ff98669e9092ce1b51a2fc97c73f31_2_690x374.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30a219f872ff98669e9092ce1b51a2fc97c73f31_2_1035x561.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/0/30a219f872ff98669e9092ce1b51a2fc97c73f31.png 2x" data-dominant-color="F6F5F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1339×726 30.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: ptrblck
cooked: <p>One potential issue would be the usage of this attached tensor:</p>
<pre><code class="lang-python">logger(loss_batch, metrics_batch, batch_idx)
</code></pre>
<p>Could you check, if this code path is called and, if so, could you detach it?</p>

username: wwaayyaaww
cooked: <p>Thank you for the response.<br>
I have detached the tensor as follows but unfortunately and the memory usage doesn’t solve.<br>
Do you see any other potential mistakes? ~</p>
<p>Would it be possible if it is caused by the use of <code>iteration tool</code>?</p>
<pre><code class="lang-auto">lfw_loader = itertools.cycle(lfw_loader)
</code></pre>
<pre><code class="lang-auto">        loss += loss_batch.detach()
        logger(loss.detach(), metrics, batch_idx)
        # if show_running:
        #    logger(loss.detach(), metrics, batch_idx)
        # else:
        #    logger(loss_batch.detach(), metrics_batch, batch_idx)
            
        gc.collect()

</code></pre>

username: ptrblck
cooked: <p>I don’t think it would be caused by <code>itertools</code>, but you could try to isolate the root cause of the increased memory usage by removing it as well as other parts of the code until the memory increase disappears.</p>