GPT2 large trains on 1 GPU but does not fit in two.
Author: aqred1
Pull request: no

Hi all,

I am training GPT2 from scratch with the following command:

`torchrun --nproc_per_node=2 --nnodes=1 ./5.run_clm-post.py  --model_name_or_path gpt2-large --train_file datasets/sample.txt --tokenizer_name myembeddings --do_train --do_eval --output_dir ./sample --evaluation_strategy epoch --num_train_epochs 100 --per_device_train_batch_size 24 --cache_dir .cache/`

When I train on a single A100, the model trains perfectly. When running on 2 GPUs (both A100s) I get the CUDA out of memory error. I tried to decrease to batch size 16 but still happens. Does this it mean that I have to go to batch size 8? Why does batch size 24 fit on a single GPU but not in two?

Below are the errors:

With batch size 16:

```
File "/path/to/miniconda3/lib/python3.6/site-packages/transformers/activations.py", line 42, in gelu_new
    return 0.5 * x * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))
RuntimeError: CUDA out of memory. Tried to allocate 320.00 MiB (GPU 0; 39.59 GiB total capacity; 36.81 GiB already allocated; 205.69 MiB free; 37.07 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

With batch size 24:
```
File "/path/to/miniconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 1169, in dropout
    return _VF.dropout_(input, p, training) if inplace else _VF.dropout(input, p, training)
RuntimeError: CUDA out of memory. Tried to allocate 1.88 GiB (GPU 1; 39.59 GiB total capacity; 36.11 GiB already allocated; 909.69 MiB free; 36.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

Any help would be appreciated. Also, any advice to make the model train faster would be great to follow.
Thanks for this great repository.

Comments:
This [really great document](https://huggingface.co/docs/transformers/performance), written by @stas00 , may be of help :)

That's very interesting. There should be a bit of memory usage overhead for many gpus over 1 gpu because the latter has to sync DDP results, but not by much.

What was the command line for a single gpu?

What was the max gpu memory usage with 1 gpu?

> any advice to make the model train faster would be great to follow.

what Lysandre said. Specifically enabling https://huggingface.co/docs/transformers/master/performance#bf16

Additionally you may want to experiment with Deepspeed Stage-2 https://huggingface.co/docs/transformers/master/main_classes/deepspeed but it doesn't yet support bfloat - should happen soon.

Hi all,

Thank you for the suggestions. I didn't know about bf16. I'll check whether that's faster than deepspeed + fp16 which is what I'm finally running at the moment. 

The command for one GPU was: 
`python ./5.run_clm-post.py --model_name_or_path gpt2-large --train_file datasets/sample.txt --tokenizer_name myembeddings --do_train --do_eval --output_dir ./sample --evaluation_strategy epoch --num_train_epochs 100 --per_device_train_batch_size 24 --cache_dir .cache/`

I have switched over to a computer with 8 A40s and I cannot reproduce the error there. But I could re-test in the A100 if you're interested. Otherwise, feel free to close the issue.

Thanks

Unfortunately, I don't currently have access to A100, as it'd be much easier for me to test directly.

So let's keep it open and once I can test I will be happy to experiment and see if there are any problems. I will assign this to me so that it doesn't get lost.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.