Memory leak
Author: yurkoff-mv
Pull request: no
Labels: name: solved

### System Info

OS: Ubuntu 20.04

- `transformers` version: 4.31.0
- Platform: Linux-5.15.0-79-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


Python packages:
torch==2.0.1+cu118; sys_platform == 'linux'
torchvision==0.15.2+cu118; sys_platform == 'linux'
torchtext==0.15.2; sys_platform == 'linux'
torchaudio==2.0.2+cu118; sys_platform == 'linux'
psutil==5.9.5
requests==2.31.0
captum==0.6.0
packaging==23.1
pynvml==11.4.1
pyyaml==6.0
nvgpu
cython==0.29.34
wheel==0.40.0
pillow==9.3.0
numpy==1.24.3
torchtext==0.15.2
torchserve==0.7.1
torch-model-archiver==0.7.1
transformers==4.31.0
tokenizers==0.13.3
sentencepiece==0.1.99
bitsandbytes==0.41.1
accelerate==0.21.0
scipy==1.10.1

### Who can help?

@sgugger, @muellerzr, @ArthurZucker and @younesbelkada

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers import LlamaTokenizerFast, LlamaForCausalLM

# model_name = 'TheBloke/Llama-2-13B-fp16'
model_name = 'lmsys/vicuna-13b-v1.5-16k'
tokenizer = LlamaTokenizerFast.from_pretrained(model_name )
model = LlamaForCausalLM.from_pretrained(model_name ,
                                         load_in_8bit=True,
                                         device_map='sequential',
                                         torch_dtype=torch.float16,
                                         low_cpu_mem_usage=True,
                                         )
```

**prompt = "LARGE PROMPT"**

```
inputs = self.tokenizer(prompts)
output_ids = self.model.generate(torch.as_tensor(inputs.input_ids).to(self.device),
                                 do_sample=True,
                                 temperature=0.8,
                                 max_new_tokens=512,
                                 top_p=0.95,
                                 # synced_gpus=True,
                                 )
results = self.tokenizer.batch_decode(output_ids,
                                      skip_special_tokens=True,
                                      clean_up_tokenization_spaces=False)[0]
```
**BREAKPOINT**

Model after loading
![image](https://github.com/huggingface/transformers/assets/82467993/29aea701-9e9d-4876-992a-05f8e05495bc)


Model in inference
![image](https://github.com/huggingface/transformers/assets/82467993/18115e2b-a9cf-4577-bd68-9c3fbbe5352e)


Model after inference
![image](https://github.com/huggingface/transformers/assets/82467993/48689f90-79c6-4a6e-963b-0387a24e6a89)

After the calculations are completed, the model does not return the used memory to the pool.

### Expected behavior

After the calculations are completed, the amount of memory occupied should return to the previous values as when loading the model (56.5 % or ~14GB).

Comments:
You need to call the garbage collector and `torch.cuda.empty_cache()` at the very least to get a chance to get the memory back.

After every inference? I didn't do this before and it was done automatically.

If you do a new inference, you won't see new memory used, just the same one being re-used. You need to properly empty cache/garbage collect when doing memory measurements.

I call 
```
gc.collect()
torch.cuda.empty_cache()
```
after each inference and the problem is gone. Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.