GPU memory isn't freed while using trainer. GPU runs out of memory and throws OOM
Author: Datta0
Pull request: no

### System Info

- OS : Ubtuntu 2204
- Cuda: 12.0
- Driver: 525.125.06
- Transformers Version: 4.33.2
- GPU: 1 x A100-40GB VRAM via PCIe

### Who can help? @muellerz @pacman100 ( Trainer is what I'm looking at)

GPU runs out of memory after a few steps ( generally around 180 to start with, then resume_from_checkpoint gets it to 250, then to 300.... and never goes past 500). Usage keeps on increasing every step.


Model: [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) Loaded in BF16.
Dataset: [OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca)
Lora Config:  Given below



### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Finetune a **llama2-7b-hf** model using **A100 40GB** on OpenOrca dataset using LoraConfig as follows:
Here's my [notebook](https://colab.research.google.com/drive/10CdhOTWUbEFQCEgD5yflfXH-Jnnsa7wU?usp=sharing) for reference ( No I didn't run on colab, but I found colab easier to share).


```
model = AutoModelForCausalLM.from_pretrained(model_name,torch_dtype = torch.bfloat16,device_map = 'auto', )

for param in model.parameters():  # Freezing original weights
  param.requires_grad = False 
  
lora_config = LoraConfig(r = 16, lora_alpha=64, target_modules=["q_proj","v_proj","o_proj","k_proj"], lora_dropout=0.1, bias = "none", task_type = "CAUSAL_LM" )
# Trainable params : 0.24% ~17 million ~ 68MB in FP32

trainer = Trainer(
    model = model,
    train_dataset = train_data,
    eval_dataset = eval_data,
    args = TrainingArguments(per_device_train_batch_size=2, gradient_accumulation_steps=4, learning_rate=2e-4, bf16=True, logging_steps=20, output_dir=output_dir, optim=f"paged_adamw_{opt_bits}bit", save_steps = 20,  save_total_limit=3, disable_tqdm = False),
    data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False)
)
model.config.use_cache = False
trainer.train(resume_from_checkpoint=resume())

```

### Expected behavior

GPU RAM should be freeed once the execution of one batch/step is done or atleast when it is needed for the later batches.

Note: If I explicitly add the following lines at the start of every step [here](https://github.com/huggingface/transformers/blob/37c205eb5d5165b70d3100b599a2bcfc483944f5/src/transformers/trainer.py#L1865) 
```
gc.collect()
trainer.accelerator.clear()
torch.cuda.empty_cache()
```
It goes on for quite a lot more steps. GPU RAM usage goes down occasionally making the finetune run longer and can be resumed later. Attaching the graph when I added "clear()" in transformers.


<img width="1022" alt="image" src="https://github.com/huggingface/transformers/assets/39181234/20cd7c4f-1652-4f7c-afc6-7044207fb108">


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Ah curious why this is closed. 
@muellerzr do I need to reopen this or is this fixed and hence closed?

@Datta0 Is this solved? I am also curious about this result.

@SangbumChoi I don't see the issue anymore. IDK what caused the issue. Also off late, I'm using [unsloth](https://github.com/unslothai/unsloth) for all my fine tunes. It is just couple of lines of change over HF code with tremendous improvements.

@Datta0 Very interesting, also thanks for sharing the new repo!