OutOfMemoryError: CUDA out of memory despite available GPU memory
Author: HumzaSami00
Pull request: no

### System Info

I’m encountering an issue with GPU memory allocation while training a GPT-2 model on a GPU with 24 GB of VRAM. Despite having a substantial amount of available memory, I’m receiving the following error:

`OutOfMemoryError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 23.68 GiB total capacity; 18.17 GiB already allocated; 64.62 MiB free; 18.60 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF.`

Here are the specifications of my setup and the model training:

GPU: NVIDIA GPU with 24 GB VRAM
Model: GPT-2 with approximately 3 GB in size and 800 parameters of 32-bit each
Training Data: 36,000 training examples with input_ids length of 600
Training Configuration: 5 epochs, batch size of 16, and fp16 enabled

These are my calculations:

Parameters: 775M parameters of 32 bits each
Gradients:
Gradients are typically of the same size as the model’s parameters.
Batch Size and Training Examples:
Batch Size: 16
Training Examples: 36,000
Vector Length: 600
Memory Allocation per Batch:
Model: 3 GB (unchanged per batch)
Gradients: 3 GB (unchanged per batch)
Input Data: 16 x 600 (vector length) x 4 bytes (assuming each value is a 32-bit float) = 37.5 KB per batch
Output Data: 16 x 600 (vector length) x 4 bytes (assuming each value is a 32-bit float) = 37.5 KB per batch
Based on the above calculations, the memory allocation per batch for my scenario would be approximately:

Model: 3 GB
Gradients: 3 GB
Input and Output Data: 75 KB
Training should not take more memory than 7GB maximum. But Its taking ~23GB of VRAM.
I would appreciate any insights or suggestions on how to resolve this issue. Thank you in advance for your assistance!



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
MODEL_NAME = "gpt2-large"

model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
tokenizer.pad_token = tokenizer.eos_token
batch = 16
epoch = 5
g_acc_step = 5
lr = 2e-6
training_args = transformers.TrainingArguments(
    per_gpu_train_batch_size=batch,
    gradient_accumulation_steps=g_acc_step,
    num_train_epochs=epoch,
    learning_rate=lr,
#     fp16=True,
    save_total_limit=1,
    logging_steps=50,
    logging_strategy = "steps",
    output_dir=OUTPUT_DIR,
    max_steps=-1,
    lr_scheduler_type="cosine",
    save_strategy ="epoch"
)


trainer = transformers.Trainer(
    model=model,
    train_dataset=filtered_dataset,
    args=training_args,
    callbacks=[LogCallback],
    data_collator=transformers.DataCollatorForLanguageModeling (tokenizer, mlm=False))

model.config.use_cache= False

```

### Expected behavior

Model is taking alot more memory than as expected

Comments:
Your computation does not inclide the optimizer states (an additional 2*3B) and all intermediate activations saved to compute the radients for the backward pass, which will be huge at a batch size of 16 with sequence lengths of 600

Seems you are right. But Why memoray allocation is keep increasing ? I started training with batch size 2 and it allocated 9GB VRAM but after 3 epoch it was taking 16Gb VRAM. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.