Cuda OOM training gpt2-xl with Trainer in multi-GPUs
Author: oalej-dv
Pull request: no
Labels: name: wontfix

name: Distributed Training / Models

name: gpt2

# ❓ Questions & Help

I am currently trying to finetune the gpt2-xl. I have 2 tesla T4 GPUs. However, I get the CUDA OOM error... when I look at the use of the gpus I see that the first one is full, but the second one still has enough space. Here is my code:

```
from transformers import TextDataset,DataCollatorForLanguageModeling, AutoTokenizer

import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

from transformers import GPT2LMHeadModel, Trainer, TrainingArguments
model = GPT2LMHeadModel.from_pretrained("gpt2-xl").to(device)

from transformers import TextDataset,DataCollatorForLanguageModeling, AutoTokenizer,  TrainingArguments, Trainer
tokenizer = AutoTokenizer.from_pretrained("gpt2-xl")
train_dataset = TextDataset(
          tokenizer=tokenizer,
          file_path='dataset_training.txt',
          block_size=128)

data_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer, mlm=False,
    )

training_args = TrainingArguments(
    output_dir='./results',         # output directory
    num_train_epochs=2,              # total # of training epochs
    per_device_train_batch_size=1,  # batch size per device during training
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs', 
    
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    prediction_loss_only=True,
)

trainer.train()
```
I get "CUDA out of memory. Tried to allocate 40.00 MiB (GPU 0; 14.73 GiB total capacity; 13.61 GiB already allocated; 31.88 MiB free; 13.98 GiB reserved in total by PyTorch)"

When I run nvidia-smi I see:

```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+================
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   75C    P0    34W /  70W |  15047MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:00:05.0 Off |                    0 |
| N/A   56C    P0    29W /  70W |   9479MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|===============================================
|    0      1756      C   /opt/conda/bin/python                      15037MiB |
|    1      1756      C   /opt/conda/bin/python                       9469MiB |
+-----------------------------------------------------------------------------+
```
**My question is:** Am I making a mistake? or how can a large model be trained with multiple GPUs?


Comments:
I want to make an update. I thought it might be possible that gpt2-xl was impossible to finu-tune, so I tested it with gpt2-large, but I got the same result: "CUDA out of memory".

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@fumpe Did you find a way around to this?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.