RuntimeError: CUDA out of memory. Tried to allocate 786.00 MiB (GPU 0; 14.73 GiB total capacity; 13.33 GiB already allocated; 575.88 MiB free; 13.38 GiB reserved in total by PyTorch)
Author: david1983
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2
Language I am using the model on (English, Chinese ...): eng

**The tasks I am working on is:**

!python /content/transformers/examples/run_language_modeling.py --train_data_file=shakespeare.txt --model_type=gpt2 --model_name_or_path=gpt2 --output_dir=output --do_train

## To reproduce

Steps to reproduce the behavior:

import os
import requests
file_name = "shakespeare.txt"
if not os.path.isfile(file_name):
	url = "https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt"
	data = requests.get(url)
	
	with open(file_name, 'w') as f:
		f.write(data.text)
!python /content/transformers/examples/run_language_modeling.py --train_data_file=shakespeare.txt --model_type=gpt2 --model_name_or_path=gpt2 --output_dir=output --do_train



03/20/2020 13:36:52 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
03/20/2020 13:36:53 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json from cache at /root/.cache/torch/transformers/4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.699bbd1c449e9861456f359d6daa51bd523ac085b4b531ab0aad5a55d091e942
03/20/2020 13:36:53 - INFO - transformers.configuration_utils -   Model config GPT2Config {
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": null,
  "do_sample": false,
  "embd_pdrop": 0.1,
  "eos_token_ids": null,
  "finetuning_task": null,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_epsilon": 1e-05,
  "length_penalty": 1.0,
  "max_length": 20,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 1024,
  "num_beams": 1,
  "num_labels": 2,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": null,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 50257
}

03/20/2020 13:36:54 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json from cache at /root/.cache/torch/transformers/f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
03/20/2020 13:36:54 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt from cache at /root/.cache/torch/transformers/d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
03/20/2020 13:36:54 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin from cache at /root/.cache/torch/transformers/4295d67f022061768f4adc386234dbdb781c814c39662dd1662221c309962c55.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1
03/20/2020 13:37:02 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, block_size=1024, cache_dir=None, config_name=None, device=device(type='cuda'), do_eval=False, do_train=True, eval_all_checkpoints=False, eval_data_file=None, evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=5e-05, line_by_line=False, local_rank=-1, logging_steps=500, max_grad_norm=1.0, max_steps=-1, mlm=False, mlm_probability=0.15, model_name_or_path='gpt2', model_type='gpt2', n_gpu=1, no_cuda=False, num_train_epochs=1.0, output_dir='output', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=4, per_gpu_train_batch_size=4, save_steps=500, save_total_limit=None, seed=42, server_ip='', server_port='', should_continue=False, tokenizer_name=None, train_data_file='shakespeare.txt', warmup_steps=0, weight_decay=0.0)
03/20/2020 13:37:02 - INFO - __main__ -   Loading features from cached file gpt2_cached_lm_1024_shakespeare.txt
03/20/2020 13:37:02 - INFO - __main__ -   ***** Running training *****
03/20/2020 13:37:02 - INFO - __main__ -     Num examples = 330
03/20/2020 13:37:02 - INFO - __main__ -     Num Epochs = 1
03/20/2020 13:37:02 - INFO - __main__ -     Instantaneous batch size per GPU = 4
03/20/2020 13:37:02 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 4
03/20/2020 13:37:02 - INFO - __main__ -     Gradient Accumulation steps = 1
03/20/2020 13:37:02 - INFO - __main__ -     Total optimization steps = 83
Epoch:   0% 0/1 [00:00<?, ?it/s]
Iteration:   0% 0/83 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/content/transformers/examples/run_language_modeling.py", line 799, in <module>
    main()
  File "/content/transformers/examples/run_language_modeling.py", line 749, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "/content/transformers/examples/run_language_modeling.py", line 353, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 602, in forward
    shift_logits = lm_logits[..., :-1, :].contiguous()
RuntimeError: CUDA out of memory. Tried to allocate 786.00 MiB (GPU 0; 14.73 GiB total capacity; 13.33 GiB already allocated; 575.88 MiB free; 13.38 GiB reserved in total by PyTorch)

Epoch:   0% 0/1 [00:00<?, ?it/s]
Iteration:   0% 0/83 [00:00<?, ?it/s]




Comments:
@mariamabarham didn't you have a similar issue? 

Yes I encountered the same issue. I solved it by adding --fp16(need to install apex first). You can also reduce the block_size to 512. Both worked out for me.

You should probably set the `per_gpu_train_batch_size` to 1. That is the default behavior for `gpt-2-simple` to prevent OOM. (I am not a fan of the default batch_size of 4 in `run_language_modeling.py`)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


You can also try using gradient accumulation steps. 
Basically, if you want a batch_size of 32, but your GPU can only fit 16.

So you make two passes of 16 batches each, accumulate your gradients, and then do the backward pass after 2 batches.