run_clm.py gpt-2 training example in documentation runs out of memory on a 32gb v100, should be verified and/or modified 
Author: HodorTheCoder
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->


- `transformers` version: 4.3.3
- Platform: Linux-4.15.0-135-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: aye!
- Using distributed or parallel set-up in script?:  no


Models:

- gpt2: @patrickvonplaten, @LysandreJik

Library:

- benchmarks: @patrickvonplaten

Documentation: @sgugger

## Information

Model I am using (Bert, XLNet ...): running the run_clm.py fine-tuning script on gpt-2

The problem arises when using:
* [x] the official example scripts: tranformers/example/language-modeling/run_clm.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)  Using the official huggingface wikitext-2-raw-v1 dataset
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Using the example here:
https://github.com/huggingface/transformers/tree/master/examples/language-modeling

When fine-tuning gpt-2 with run_clm.py, this should run on a k80 (24gb of RAM) in about an hour according to the example.  However, I'm running out of memory with default settings... and I'm using a v100 inside a DXG-9, with 32gb of memory:

```
nvidia-smi
Tue Mar 16 15:19:23 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:06:00.0 Off |                    0 |
| N/A   34C    P0    44W / 300W |      0MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
```

Pasting the command here for clarity.

```
python run_clm.py \
    --model_name_or_path gpt2 \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --do_train \
    --do_eval \
    --output_dir /tmp/test-clm
```
and the eponymous error:

`RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 31.72 GiB total capacity; 30.32 GiB already allocated; 187.88 MiB free; 30.38 GiB reserved in total by PyTorch)`

That's a ton of memory... is this right?  Or is there some type of memory leak? 

Now, this can be fixed by setting `--per_device_train_batch_size 4` but I highly doubt the current format will work out of the box on a k80 w/out changing anything (which I can't test because I don't have access to one) using the default batch size of `8`, and this should be reflected in the example and/or verified with the current `run_clm.py`.

Now, on a v100, it did finish in a little under 14 minutes, which is incredibly fast-- so I'm not complaining-- but I know batch size should be as high as possible on these things to get the best results, and I was really hoping it would work with 8 (in fact I was hoping I could jack it up to 16 by wrapping it in nn.torchDataParallel, but that's for another day.)

This leads me to another question-- I know you can do torch.distributed.launch with these scripts, is there one that wraps the model in `nn.parallel.DistrubtedDataParallel` so that you can chunk a larger batch size across multiple GPU's and utilize the extra memory, or should this be done by hand?  If so, maybe I will create a PR and add an option for this inside the three  example scripts as it would be quite beneficial.  Example:

```
model = torch.nn.parallel.DistributedDataParallel(model,
                                                      device_ids=[args.local_rank],
                                                      output_device=args.local_rank)
```

Results:

```
Training completed. Do not forget to share your model on huggingface.co/models =)



{'train_runtime': 820.2706, 'train_samples_per_second': 2.121, 'epoch': 3.0}                                                                                       
100%|##########################################################################################################################| 1740/1740 [13:40<00:00,  2.12it/s]
[INFO|trainer.py:1408] 2021-03-16 15:39:31,084 >> Saving model checkpoint to /tmp/test-clm
[INFO|configuration_utils.py:304] 2021-03-16 15:39:31,085 >> Configuration saved in /tmp/test-clm/config.json
[INFO|modeling_utils.py:817] 2021-03-16 15:39:32,049 >> Model weights saved in /tmp/test-clm/pytorch_model.bin
03/16/2021 15:39:32 - INFO - __main__ -   ***** Train results *****
03/16/2021 15:39:32 - INFO - __main__ -     epoch = 3.0
03/16/2021 15:39:32 - INFO - __main__ -     train_runtime = 820.2706
03/16/2021 15:39:32 - INFO - __main__ -     train_samples_per_second = 2.121
03/16/2021 15:39:32 - INFO - __main__ -   *** Evaluate ***
[INFO|trainer.py:1600] 2021-03-16 15:39:32,118 >> ***** Running Evaluation *****
[INFO|trainer.py:1601] 2021-03-16 15:39:32,118 >>   Num examples = 240
[INFO|trainer.py:1602] 2021-03-16 15:39:32,119 >>   Batch size = 8
100%|##############################################################################################################################| 30/30 [00:08<00:00,  3.52it/s]
03/16/2021 15:39:40 - INFO - __main__ -   ***** Eval results *****
03/16/2021 15:39:40 - INFO - __main__ -     perplexity = 20.967772820757663
```


Comments:
The doc has not been updated since a while ago so it's probably not up to date yes. I think the corresponding script probably had different defaults in earlier versions (either a shorter sequence length or a shorter batch size).

As for your second question, the script does work with torch.distributed.launch without changes. See the [main examples README](https://github.com/huggingface/transformers/tree/master/examples#distributed-training-and-mixed-precision) for more information.

OK.  Thanks for validating, that's kind of what I figured.

Secondly, thank you-- I think I was expecting it to split the batch for me.  So in the example on the page you sent:

```
python -m torch.distributed.launch \
    --nproc_per_node 8 text-classification/run_glue.py \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --task_name mnli \
    --do_train \
    --do_eval \
    --max_seq_length 128 \
    --per_device_train_batch_size 8 \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir /tmp/mnli_output/
```

that would be the equivalent of running a total batch size of 64 on a single GPU?

(`per_device_train_batch_size=8`) * (`nproc_per_node=8`) = `64`

Much appreciated.

Yes, that's correct!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.