run_clm example gives `CUDA out of memory. Tried to allocate` error
Author: massanishi
Pull request: no

## Environment info

Google Colab with GPU runtime.

- Python version: 3.6.9

## Information

I'm trying to run the GPT2 training example from `https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py`.

The problem arises when using:

- Run CLM language modeling example.

## To reproduce

Steps to reproduce the behavior:

1. Open Google Colab with GPU on

2. Run

```
!git clone https://github.com/huggingface/transformers
%cd transformers
!pip install .
%cd examples
%cd language-modeling
!pip install -r requirements.txt
```

```
!python run_clm.py \
    --model_name_or_path gpt2 \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --do_train \
    --do_eval \
    --output_dir test
```

Log:

```
[INFO|trainer.py:668] 2020-12-14 02:09:02,049 >> ***** Running training *****
[INFO|trainer.py:669] 2020-12-14 02:09:02,049 >>   Num examples = 2318
[INFO|trainer.py:670] 2020-12-14 02:09:02,049 >>   Num Epochs = 3
[INFO|trainer.py:671] 2020-12-14 02:09:02,049 >>   Instantaneous batch size per device = 8
[INFO|trainer.py:672] 2020-12-14 02:09:02,049 >>   Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:673] 2020-12-14 02:09:02,049 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:674] 2020-12-14 02:09:02,049 >>   Total optimization steps = 870
  0% 0/870 [00:00<?, ?it/s]Traceback (most recent call last):
  File "run_clm.py", line 357, in <module>
    main()
  File "run_clm.py", line 327, in main
    trainer.train(model_path=model_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 767, in train
    tr_loss += self.training_step(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1096, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1120, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 895, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 740, in forward
    output_attentions=output_attentions,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 295, in forward
    output_attentions=output_attentions,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 239, in forward
    attn_outputs = self._attn(query, key, value, attention_mask, head_mask, output_attentions)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 166, in _attn
    w = torch.matmul(q, k)
RuntimeError: CUDA out of memory. Tried to allocate 384.00 MiB (GPU 0; 15.90 GiB total capacity; 14.75 GiB already allocated; 185.88 MiB free; 14.81 GiB reserved in total by PyTorch)
  0% 0/870 [00:00<?, ?it/s]
```

## Expected behavior

Outputs the model in the output_dir with no memory error.


Comments:
You should try to reduce the batch size. This will reduce the memory usage.

Yup. What @LysandreJik said is correct. Use the following:
`--per_device_train_batch_size x \`
`--per_device_eval_batch_size x \`
Replace x with your preferred batch size, I would recommend the highest power of 2 your GPU memory allows.

It worked! With the Colab's GPU memory size of 12.72GB, the batch size worked at:

`--per_device_train_batch_size 2 \`
`--per_device_eval_batch_size 16 \`

Thanks for the quick response guys.