How to remove variables which are converted to gpu tensors inline?

username: shifathosn
cooked: <p>Hi there. I am trying to run a simple model by giving some input data to it. The model tensors are in cuda device. The way I am feeding the data to the model is by converting the cpu tensor to the cuda tensor by inline conversion right at the model input <code>output = model(input.cuda())</code>.</p>
<p>But after running this line I could see that the used memory of the gpu is increased (which is very normal). But after running the same line again, the newly created input gpu tensor (with the inline conversion) is again allocated to the gpu, which I guess is not normal.</p>
<p>In this case, is there any way to remove the previously created gpu tensor and deallocate gpu memory? Is there any way to find the variable handles which are created inline as shown in the example?</p>
<p>N.B. I know I could convert the cpu tensors to a gpu tensor and store in a variable to retain the variable handle (<code>gpuTensor = cpuTensor.cuda()</code>). And then when the work is done I could simply run <code>del gpuTensor</code>. That would deallocate the gpu memory. But the question is not how to remove a gpu tensor.</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>PyTorch will move the memory back to the cache once all references to the tensor are gone. How did you verify that new memory is allocated for the temp input?</p>

username: shifathosn
cooked: <p>I have used <code>nvidia-smi</code> and <code>nvidia-htop.py</code> to see the allocated memory. My basic understanding is: re-running a code block with the same variable alias should not occupy more memory as the previous tensor will lose reference to a variable. This should also be true for temporary allocated memory blocks.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="shifathosn" data-post="3" data-topic="196334">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shifathosn/48/67335_2.png" class="avatar"> shifathosn:</div>
<blockquote>
<p>re-running a code block with the same variable alias should not occupy more memory as the previous tensor will lose reference to a variable.</p>
</blockquote>
</aside>
<p>That’s right and is also the case as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch


print(torch.cuda.memory_allocated() / 1024**2)
# 0.0
print(torch.cuda.memory_reserved() / 1024**2)
# 0.0

def fun():
    # create 5MB tensor
    x = torch.randn(5 * 1024**2 // 4, device="cuda")
    print("inside ", torch.cuda.memory_allocated() / 1024**2)
    print("inside ", torch.cuda.memory_reserved() / 1024**2)
    
for _ in range(5):
    fun()
    print("outside ", torch.cuda.memory_allocated() / 1024**2)
    print("outside ", torch.cuda.memory_reserved() / 1024**2)
    print("="*10)

# inside  5.0
# inside  20.0
# outside  0.0
# outside  20.0
# ==========
# inside  5.0
# inside  20.0
# outside  0.0
# outside  20.0
# ==========
# inside  5.0
# inside  20.0
# outside  0.0
# outside  20.0
# ==========
# inside  5.0
# inside  20.0
# outside  0.0
# outside  20.0
# ==========
# inside  5.0
# inside  20.0
# outside  0.0
# outside  20.0
# ==========
</code></pre>
<p><code>nvidia-smi</code> will show the GPU memory usage of all processes and thus also from the CUDA context etc., so is not trivial to map to tensor data.<br>
As you can see in my example, the temp. tensor will use memory while it’s inside the function scope (seen via <code>torch.cuda.memory_allocated()</code>) and its memory will be released again in the for loop. The cache is allocated to 20MB and will be reused.</p>