Memory growth issues in custom architecture

ANDERSON_SOUZA: <p>Hi everyone.</p>
<p>I’m training a custom architecture for a specific problem. This architecture works perfectly. However, I’m trying to run an experiment to calculate the loss according to the attributes. In other words, I select the attribute and calculate the loss for the selected attribute.</p>
<p>The problem that is occurring is the increase in memory. I have tried to analyze what is happening, but I couldn’t figure it out.</p>
<p>Here is part of my code.</p>
<pre><code>prev_size = 1
for attribute_item in combined_attributes:
    NUM_FEATURES = len(attribute_item)
    print(attribute_item)
if NUM_FEATURES != prev_size:       
    reset_model_optimizer()
    prev_size = NUM_FEATURES

start_time = time.time()
for epoch in range(50):
    # Create a generator object for data batches
    data_gen = data_generator(trainloader)

    for i, examples in enumerate(data_gen, 0):
        (inputs, desired_output) = examples

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward
        output_predictions = model(inputs)

        # loss
        desired_output = desired_output.to(output_predictions.device)
        loss = torch.nn.MSELoss()(output_predictions, desired_output)

        # backward + optimize
        loss.backward()
        optimizer.step()

        # New 
        output_loss.append(loss.item())

output_attribute.append(attribute_item)
output_epoch.append(j for j in range(50))

end_time = time.time()
execution_time = end_time - start_time
minutes = int(execution_time // 60)
seconds = int(execution_time % 60)
time_format = "{:02d}:{:02d}".format(minutes, seconds)
print("Time elapsed:", time_format)

#df_tmp = pd.DataFrame()
#df_tmp['Attribute'] = output_attribute
#df_tmp['Epoch'] = output_epoch
#df_tmp['Loss'] =  output_loss

#file_name = '__'.join(attribute_item) + ".xlsx" 
#df_tmp.to_excel(file_name, index=False)

output_attribute = []
output_epoch = []
output_loss = []

del inputs  
del desired_output
del output_predictions
del loss
</code></pre>
<p>If necessary, here is the complete code is here:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/11U-dsEseSzaNDEIubXXtXcNEE2jWQ2Gs?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/8f7f9e638c98b96f300eabeb3c2051a6/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/11U-dsEseSzaNDEIubXXtXcNEE2jWQ2Gs?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/11U-dsEseSzaNDEIubXXtXcNEE2jWQ2Gs?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I would appreciate it if someone could help me, please.</p>
<p>Thank you very much.</p>