Overwriting Intermediate Variables and Memory Usage

username: lerner
cooked: <p>Does overwriting the value of existing variables free memory for possible garbage collection?</p>
<p>Under <code>torch.no_grad()</code> I assume the tensors’ memory is free for collection, as a computation graph is not stored. But presumably if tracking gradients each intermediate tensor is stored in the computation graph for the backwards pass, regardless of whether you overwrite the references to them?</p>
<p>For example, where <code>x</code> is a tensor requiring gradients and <code>{f_i}</code> differentiable operations, we can store the intermediate values,</p>
<pre data-code-wrap="python"><code class="lang-python">x = f(x)
x1 = f1(x)
x2 = f2(x1)
</code></pre>
<p>or we can overwrite the intermediate values.</p>
<pre data-code-wrap="python"><code class="lang-python">x = f(x)
x = f1(x)
x = f2(x)
</code></pre>

username: soulitzer
cooked: <p>Overwriting intermediates is generally preferable, as not all intermediates are saved for backward<br>
e.g. x.sin().exp(), the result of sin is not saved. Also consider cases when the op is linear, or when activation checkpoint is used.</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/soulitzer">@soulitzer</a> <a class="mention" href="/u/lerner">@lerner</a>  just for fun, i tried checking the memory before and after. Memory of the process did not change … maybe because the stack memory which contains references is still containing the same amount of references</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.linkedin.com/pulse/memory-management-python-concept-heap-stack-garbage-collector-usman/">
  <header class="source">
      <img src="https://static.licdn.com/aero-v1/sc/h/al2o9zrvru7aqj8e1x2rzsrca" class="site-icon" width="64" height="64">

      <a href="https://www.linkedin.com/pulse/memory-management-python-concept-heap-stack-garbage-collector-usman/" target="_blank" rel="noopener nofollow ugc">linkedin.com</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:666/500;"><img src="https://media.licdn.com/dms/image/D4E12AQFLvY0tTE6XSg/article-cover_image-shrink_720_1280/0/1671113390582?e=2147483647&amp;v=beta&amp;t=bGUpToa4POFfxFXYzNFzhFRi3Mqu8dAW4ftsNoCmvdU" class="thumbnail" width="666" height="500"></div>

<h3><a href="https://www.linkedin.com/pulse/memory-management-python-concept-heap-stack-garbage-collector-usman/" target="_blank" rel="noopener nofollow ugc">Memory Management in Python — Concept of Heap, Stack, Garbage Collector,...</a></h3>

  <p>What will we learn from this article: Importance of Memory Management in Python The internal working mechanism of Memory (Heap, Stack) What are Reference cycles, and how does python detect these reference cycles? Garbage Collector Algorithm...</p>

  <p>
    <span class="label1">Reading time: 6 min read</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">import torch
from torch import nn
import gc
import sys
import psutil
import ctypes

class SampleModel(torch.nn.Module):
    def __init__(self):
        super(SampleModel, self).__init__()
        self.model1 = nn.Linear(10, 5)
        self.model2 = nn.Linear(10, 5)

    def forward(self, x):
        return self.model1(x)

model = SampleModel()
with torch.no_grad():
    model.to('cpu')
    print(id(list(model.model1.parameters())[0]))
    print(id(list(model.model1.parameters())[1]))
    print(id(list(model.model2.parameters())[0]))
    old_weight_address = id(list(model.model2.parameters())[0])
    old_bias_address = id(list(model.model2.parameters())[1])
    print(id(list(model.model2.parameters())[1]))
    process = psutil.Process()
    print("Memory used {}".format(process.memory_info()))
    #list(model.model2.parameters())[0] = list(model.model1.parameters())[0]
    #list(model.model2.parameters())[1] = list(model.model1.parameters())[1]
    model.model2.weight = model.model1.weight
    model.model2.bias = model.model1.bias
    print(id(list(model.model1.parameters())[0]))
    print(id(list(model.model1.parameters())[1]))
    print(id(list(model.model2.parameters())[0]))
    print(id(list(model.model2.parameters())[1]))
    gc.collect()
    print("Memory used {}".format(process.memory_info()))
</code></pre>