CPU memory (VRAM) not released after loading model in GPU
Author: arsalan993
Pull request: no

I using the latest PyTorch version with Cuda 11. 
While I load GPT2 models using "Cuda" as a device. It works perfectly fine and is able to compute on GPU but at the same time, I see it also consuming 1.5 VRAM (CPU RAM) compare to the memory it is occupying in GPU RAM.

Is it possible that once the model is loaded onto the GPU RAM we can then release the CPU VRAM?
I tried gc.collect() but it didn't work.

Comments:
Similar issue #13208.

Sorry to say.. None of the thread contains solution to this problem.. Do you guys have the solution.. So many people are asking for same problem but not a single solution is present.
Upload A GPT2-medium model to cpu first and then to GPU and you will see the VRAM is not released instead more VRAM is occupied

Arsalan Ahmed
________________________________
From: Li-Huai (Allan) Lin ***@***.***>
Sent: Friday, September 17, 2021 7:40:08 PM
To: huggingface/transformers ***@***.***>
Cc: Arsalan Ahmed ***@***.***>; Author ***@***.***>
Subject: Re: [huggingface/transformers] CPU memory (VRAM) not released after loading model in GPU (#13624)


Similar issue #13208<https://github.com/huggingface/transformers/issues/13208>.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/issues/13624#issuecomment-921851272>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFHYIGBOC5T2MT2HHTBVUZTUCNHMRANCNFSM5EGYGYGQ>.


Hello @arsalan993, could you please complete the issue template with environment information + reproducible code example so that we may take a look. Thank you.

@LysandreJik  @qqaatw 

- `transformers` version: transformers  4.10.2
- Platform: Ubuntu 18
- Python version: 3.8
- PyTorch version (GPU?):   torch==1.9.0+cu111 torchvision==0.10.0+cu111
- Tensorflow version (GPU?): None
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: No

I have 23 GB VRAM, 1080Ti 12GB graphic card
```
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Nov_30_19:08:53_PST_2020
Cuda compilation tools, release 11.2, V11.2.67
Build cuda_11.2.r11.2/compiler.29373293_0
```
---------------

Now here is the breakdown
when I upload the model on CPU then this much CPU VRAM is consumes

```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch
torch_device = 'cpu'
model = GPT2LMHeadModel.from_pretrained("gpt2-medium").to(torch_device)  
tokenizer = GPT2Tokenizer.from_pretrained("gpt2-medium")
```
CPU VRAM consumption : `2.00 GB`

---------------------------------

Now what happens which is basically an issue that I am unable to understand is if I upload the model on GPU then CPU VRAM consumption should be the least but that is not the case instead it increases

```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch
torch_device = 'cuda'
model = GPT2LMHeadModel.from_pretrained("gpt2-medium").to(torch_device)  
tokenizer = GPT2Tokenizer.from_pretrained("gpt2-medium")
```

CPU VRAM consumption : `2.58 GB`
GPU RAM consumption : `2215MiB / 11178MiB`
I was expecting it to take a few hundred. MBs of VRAM but instead VRAM consumption increased.

--------------------

I have tried the following solution https://github.com/huggingface/transformers/pull/11736 but it didn't work and increased CPU VRAM consumption a bit more.


Guys, can you please let me know maybe I am carrying some wrong concept regarding how backend architecture works. for example, if something is to be computed on GPU and a copy of it is stored on CPU VRAM.

please clear me out

@arsalan993 could you take a look at the comments in https://github.com/huggingface/transformers/issues/13208?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.