How to delete a Tensor in GPU to free up memory

ztf-ucas: <h2>How to delete a Tensor in GPU to free up memory？</h2>
<p>I can get a Tensor in GPU by Tensor.cuda(), but it just returns a copy in GPU. I wonder how can I delete this Tensor in GPU? I try to delete it with “del Tnesor” but it doesn’t work.</p>

smth: <p><code>del Tensor</code> will delete it from GPU memory. Why do you think it doesn’t work?</p>

ztf-ucas: <p>Thank you very much!<br>
I loaded an OrderedDict of pre-trained weights to gpu by torch.load(), then used a for loop to delete its elements, but there was no change in gpu memory.<br>
Besides, it is strange that there was no change in gpu memory even I deleted the OrderedDict of pre-trained weights.<br>
Pytorch version is 0.4.0.</p>

xuhdev: <p>I guess what you have missed here is <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.empty_cache" rel="nofollow noopener">torch.cuda.empty_cache</a>. After <code>del Tensor</code>, call <code>torch.cuda.empty_cache()</code> and see whether GPU memory usage changes.</p>

ztf-ucas: <p>Thank you very much！<br>
There is no change in gpu memory after excuting <code>torch.cuda.empty_cache()</code>.<br>
I just want to manually delete some unused variables such as <code>grads</code> or other intermediate variables to free up gpu memory. So I tested it by loading the pre-trained weights to gpu, then try to delete it. I’ve tried <code>del</code>, <code>torch.cuda.empty_cache()</code>, but nothing was happening.</p>

xuhdev: <p>Could you show a minimum example? The following code works for me for PyTorch 1.1.0:</p>
<pre><code>import torch
a = torch.zero(300000000, dtype=torch.int8, device='cuda')
b = torch.zero(300000000, dtype=torch.int8, device='cuda')
# Check GPU memory using nvidia-smi
del a
torch.cuda.empty_cache()
# Check GPU memory again</code></pre>

ztf-ucas: <p>Thank you very much！I guess this problem is caused by Pytorch version. I will change the version to 1.1.0.</p>

ztf-ucas: <p>Hi! I test the code you provided. I found no <code>torch.zero</code> (should be <code>torch.zeros</code>). But I found the gpu memory was not change even for Pytorch 1.1.0. This is very strange. By the way, I run this code in a docker container.</p>

woolpeeker: <p>I found that when I del the tensor, the GPU still has a small amount of memory occupation.<br>
The codes are below:</p>
<blockquote>
<p>import torch<br>
t = torch.zeros([1024, 1024, 1024, 2], device=‘cuda:0’)<br>
del t<br>
torch.cuda.empty_cache()</p>
</blockquote>
<p>The GPU still has about 700M usage<br>
|==================+=============+==================|<br>
|   0  GeForce RTX 208…  Off  | 00000000:05:00.0 Off |                  N/A |<br>
| 33%   49C    P8    29W / 250W |    717MiB / 11016MiB |      0%      Default |<br>
±------------------------------±---------------------±--------------------------------+</p>

ptrblck: <p>These 700MB of device memory are used by the CUDA context for the CUDA kernels in PyTorch as well as other libs such as cudnn, NCCL etc. and cannot be freed.</p>

Flock1: <p>Correct me if I’m wrong but I load an image and convert it to torch tensor and cuda(). So when I do that and run <code>torch.cuda.memory_allocated()</code>, it goes from 0 to some memory allocated. But then, I delete the image using <code>del</code> and then I run <code>torch.cuda.reset_max_memory_allocated()</code> and <code>torch.cuda.empty_cache()</code>, I see no change in <code>torch.cuda.memory_allocated()</code>. What should I do?</p>

ptrblck: <p>That’s the right approach, which also works for me:</p>
<pre><code class="lang-python">path = '...'
image = Image.open(path)

print(torch.cuda.memory_allocated())
&gt; 0
print(torch.cuda.memory_reserved())
&gt; 0

x = transforms.ToTensor()(image)
print(torch.cuda.memory_allocated())
&gt; 0
print(torch.cuda.memory_reserved())
&gt; 0

x = x.cuda()
print(torch.cuda.memory_allocated())
&gt; 23068672
print(torch.cuda.memory_reserved())
&gt; 23068672

del x
print(torch.cuda.memory_allocated())
&gt; 0
print(torch.cuda.memory_reserved())
&gt; 23068672

torch.cuda.empty_cache()
print(torch.cuda.memory_allocated())
&gt; 0
print(torch.cuda.memory_reserved())
&gt; 0
</code></pre>

Flock1: <p>I don’t know how but now it works. Pytorch is full of surprises.</p>
<p>One question though. Why does the memory still show up in <code>nvidia-smi</code>? Will this affect me overall memory usage?</p>

ptrblck: <p>The first CUDA operation will create the CUDA context containing the native CUDA kernels, cudnn etc. and will use the memory until the application is closed.</p>

Joseph_Summerhays: <p>I had the same problem with <code>del x</code> and <code>torch.cuda.empty_cache()</code> not removing everything off the GPU. Eventually I wrapped the for loop in <code>with torch.no_grad()</code> and now it works. I think if the gradient is turned on, it saves intermediate steps, even if you delete the final product. Therefore, turning off the gradient should solve (some people’s) problems.</p>

Aniruddha_Kalburgi: <p>Yes, either of these will do the job:</p>
<blockquote>
<p>with torch.no_grad():</p>
</blockquote>
<p>Or</p>
<blockquote>
<p>x.detach().cpu()</p>
</blockquote>

Kabir_Nagrecha: <p>Once the loop is done (say with no_grad on), is there a way to iterate through and delete these intermediate computations?</p>

sekigh: <p>Hi,</p>
<p>I still see the memory capacity insistently remains unchanged upon the following two different codes;</p>

x.detach(),
del x,
torch.cuda.empty_cache()
<p>I checked the attribute of x right after x.detach().cpu() and x still has</p>
<p>is_cuda True,<br>
grad_fun SelectBackward,<br>
requires_grad True</p>

So the second code is with replacing x.detach().cpu() with x=  x.detach().cpu() and see the x attributes right after x=  x.detach().cpu() as follows;
<p>is_cuda False,<br>
grad_fun None,<br>
requires_grad False</p>
<p>Even requires_grad and backward related attributes go away successfully, x still remains on gpu memory. Is there any insight to delet x out of gpu memory ?</p>
<p>I am hesitated to use with torch.no_grad(): because I need a few variables in a loop to have requires_grad True for backward operartion.</p>
<p>Thank you in advance.</p>

origin_of_symmetry: <p>Same problems here. Try to convert a fp32 tensor to fp16 tensor with tensor.half(), and deleting the original fp32 tensor from memory. None of these codes work.</p>

ptrblck: <p>It should work as described and verified <a href="https://discuss.pytorch.org/t/convert-float32-to-float16-with-reduced-gpu-memory-cost/128117/2">here</a>. Could you post an executable code snippet, which shows that it’s not working as intended, please?</p>