Does cuda cache memory for future usage?

username: lisyuan
cooked: <p>I found some issue in cuda memory allocation when I follow the tutorial official website guideline</p>
<pre><code class="lang-auto">device = torch.device("cuda:0")
x = torch.ones(2, 2, requires_grad=True, device =device)
print(x)
</code></pre>
<p>In this situation, <em><strong>the memory in gpu:0 is 863MB only when I create a 2 by 2 tensor array <code>x</code>.</strong></em></p>
<pre><code class="lang-auto">tensor([[1., 1.],
        [1., 1.]], device='cuda:0', requires_grad=True)
</code></pre>
<p>so, I tried to add another tensor <code>y</code> to watch the change of memory</p>
<pre><code class="lang-auto">y = x + 2
print(y)
</code></pre>
<p><em><strong>But the memory in gpu:0 is still 863MB</strong></em>. Here you can see it really track the computation history in <code>grad_fn</code></p>
<pre><code class="lang-auto">tensor([[3., 3.],
        [3., 3.]], device='cuda:0', grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>and I also try to test <code>with torch.no_grad():</code> mention in <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html</a> because it can prevent tracking history and using memory.</p>
<pre><code class="lang-auto">with torch.no_grad():
    z = y * y * 3
    out = z.mean()
    print(z)
</code></pre>
<p>here we can see the <code>grad_fn</code> in <code>z</code> disappear, so it would save the memory in gpu:0 theoretically. <em><strong>In this situation, the memory is still 863MB. Therefore, I can’t watch whether it would save memory exactly or not.</strong></em></p>
<pre><code class="lang-auto">tensor([[27., 27.],
        [27., 27.]], device='cuda:0')
</code></pre>
<p><strong>In conclusion,</strong></p>
<p><strong>1. What is the cuda memory allocation method in pytorch? Will it cache some memory for future usage? If so, how can I remove this mechanism? I don’t want to build a small model but take a lot of memory in  future.</strong></p>
<p><strong>2. Do <code>with torch.no_grad():</code> really prevent using memory? I can’t watch whether it would save memory or not in this situation.</strong></p>

username: ptrblck
cooked: <ol>
<li>
<p>The majority of the mentioned used memory comes from the creation of the CUDA context, which cannot be freed. You should also be able to see this memory usage by creating an empty <code>CUDATensor</code>.<br>
The caching allocator will reserve some memory which will be used later and you cannot disable it at the moment. Using the caching allocator will avoid synchronizing memory allocations during the model execution and should therefore speedup the execution. To check the cached memory, you could use <code>print(torch.cuda.memory_reserved())</code>. <a href="https://discuss.pytorch.org/t/clearing-the-gpu-is-a-headache/84762/4">This post</a> has a small example to show the difference between allocated and reserved memory.</p>
</li>
<li>
<p>Yes, <code>no_grad()</code> will avoid storing intermediate gradients, which would be needed during the backward pass to calculate the gradients. The memory difference of the small tensors in your example might not change the used memory in <code>nvidia-smi</code>, so you could increase the size and use <code>torch.cuda.memory_allocated</code> to check the difference.</p>
</li>
</ol>

username: lisyuan
cooked: <p>Thanks a lot! You explained it very clearly!</p>