run_summarization: fine tuning Pegasus large, CUDA out of memory error 
Author: karrtikiyer
Pull request: no

## Environment info
- `transformers` version: 4.5.0.dev0
- Platform: Linux-4.14.225-121.362.amzn1.x86_64-x86_64-with-glibc2.9
- Python version: 3.6.13
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Directly running the example script `run_summarization.py`
- Using distributed or parallel set-up in script?: Nothing explicitly set, directly running `run_summarization`

### Who can help
@patil-suraj 



## Information

Model I am using (Bert, XLNet ...): `google/pegasus-large`

The problem arises when using:
* [X ] the official example scripts: `[run_summarization.py](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_summarization.py)`

The tasks I am working on is:
* [X ] my own task or dataset: I am trying to fine tune pegasus-large on my custom data set of articles and their summary. I have 100 pairs of documents and summaries in train set while I have around 48 in the validation set. The input is fed as CSV file.

## To reproduce

Steps to reproduce the behavior:

1. Run the file `run_summarization.py` with below parameters:
```
 python run_summarization.py \
    --model_name_or_path google/pegasus-large \
    --do_train \
    --do_eval \
    --train_file /data/train_file.csv \
    --validation_file data/test_file.csv \
    --output_dir /output \
    --overwrite_output_dir \
    --per_device_train_batch_size=1 \
    --per_device_eval_batch_size=1 \
    --predict_with_generate
```
2. It errors when the training process starts:
```
python run_summarization.py \
>     --model_name_or_path google/pegasus-large \
>     --do_train \
>     --do_eval \
>     --train_file /data/train_file.csv \
>     --validation_file /data/test_file.csv \
>     --output_dir /output \
>     --overwrite_output_dir \
>     --per_device_train_batch_size=1 \
>     --per_device_eval_batch_size=1 \
>     --predict_with_generate
04/05/2021 11:17:50 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 8distributed training: False, 16-bits training: False
04/05/2021 11:17:50 - INFO - __main__ -   Training/evaluation parameters Seq2SeqTrainingArguments(output_dir='/output', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=False, evaluation_strategy=<IntervalStrategy.NO: 'no'>, prediction_loss_only=False, per_device_train_batch_size=1, per_device_eval_batch_size=1, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, lr_scheduler_type=<SchedulerType.LINEAR: 'linear'>, warmup_ratio=0.0, warmup_steps=0, logging_dir='runs/Apr05_11-17-50_ip-172-16-93-105', logging_strategy=<IntervalStrategy.STEPS: 'steps'>, logging_first_step=False, logging_steps=500, save_strategy=<IntervalStrategy.STEPS: 'steps'>, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', fp16_backend='auto', fp16_full_eval=False, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name='/output', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, sharded_ddp=[], deepspeed=None, label_smoothing_factor=0.0, adafactor=False, group_by_length=False, length_column_name='length', report_to=[], ddp_find_unused_parameters=None, dataloader_pin_memory=True, skip_memory_metrics=False, mp_parameters='', sortish_sampler=False, predict_with_generate=True)
04/05/2021 11:17:50 - WARNING - datasets.builder -   Using custom data configuration default-511203313715199c
04/05/2021 11:17:50 - WARNING - datasets.builder -   Reusing dataset csv (.cache/huggingface/datasets/csv/default-511203313715199c/0.0.0/2dc6629a9ff6b5697d82c25b73731dd440507a69cbce8b425db50b751e8fcfd0)
loading configuration file https://huggingface.co/google/pegasus-large/resolve/main/config.json from cache at .cache/huggingface/transformers/3fa0446657dd3714a950ba400a3fa72686d0f815da436514e4823a973ef23e20.7a0cb161a6d34c3881891b70d4fa06557175ac7b704a19bf0100fb9c21af9286
Model config PegasusConfig {
  "_name_or_path": "google/pegasus-large",
  "activation_dropout": 0.1,
  "activation_function": "relu",
  "add_bias_logits": false,
  "add_final_layer_norm": true,
  "architectures": [
    "PegasusForConditionalGeneration"
  ],
  "attention_dropout": 0.1,
  "bos_token_id": 0,
  "classif_dropout": 0.0,
  "classifier_dropout": 0.0,
  "d_model": 1024,
  "decoder_attention_heads": 16,
  "decoder_ffn_dim": 4096,
  "decoder_layerdrop": 0.0,
  "decoder_layers": 16,
  "decoder_start_token_id": 0,
  "dropout": 0.1,
  "encoder_attention_heads": 16,
  "encoder_ffn_dim": 4096,
  "encoder_layerdrop": 0.0,
  "encoder_layers": 16,
  "eos_token_id": 1,
  "extra_pos_embeddings": 1,
  "force_bos_token_to_be_generated": false,
  "forced_eos_token_id": 1,
  "gradient_checkpointing": false,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2"
  },
  "init_std": 0.02,
  "is_encoder_decoder": true,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2
  },
  "length_penalty": 0.8,
  "max_length": 256,
  "max_position_embeddings": 1024,
  "model_type": "pegasus",
  "normalize_before": true,
  "normalize_embedding": false,
  "num_beams": 8,
  "num_hidden_layers": 16,
  "pad_token_id": 0,
  "scale_embedding": true,
  "static_position_embeddings": true,
  "task_specific_params": {
    "summarization_aeslc": {
      "length_penalty": 0.6,
      "max_length": 32,
      "max_position_embeddings": 512
    },
    "summarization_arxiv": {
      "length_penalty": 0.8,
      "max_length": 256,
      "max_position_embeddings": 1024
    },
    "summarization_big_patent": {
      "length_penalty": 0.7,
      "max_length": 256,
      "max_position_embeddings": 1024
    },
    "summarization_billsum": {
      "length_penalty": 0.6,
      "max_length": 256,
      "max_position_embeddings": 1024
    },
    "summarization_cnn_dailymail": {
      "length_penalty": 0.8,
      "max_length": 128,
      "max_position_embeddings": 1024
    },
    "summarization_gigaword": {
      "length_penalty": 0.6,
      "max_length": 32,
      "max_position_embeddings": 128
    },
    "summarization_large": {
      "length_penalty": 0.8,
      "max_length": 256,
      "max_position_embeddings": 1024
    },
    "summarization_multi_news": {
      "length_penalty": 0.8,
      "max_length": 256,
      "max_position_embeddings": 1024
    },
    "summarization_newsroom": {
      "length_penalty": 0.8,
      "max_length": 128,
      "max_position_embeddings": 512
    },
    "summarization_pubmed": {
      "length_penalty": 0.8,
      "max_length": 256,
      "max_position_embeddings": 1024
    },
    "summarization_reddit_tifu": {
      "length_penalty": 0.6,
      "max_length": 128,
      "max_position_embeddings": 512
    },
    "summarization_wikihow": {
      "length_penalty": 0.6,
      "max_length": 256,
      "max_position_embeddings": 512
    },
    "summarization_xsum": {
      "length_penalty": 0.8,
      "max_length": 64,
      "max_position_embeddings": 512
    }
  },
  "transformers_version": "4.5.0.dev0",
  "use_cache": true,
  "vocab_size": 96103
}

loading configuration file https://huggingface.co/google/pegasus-large/resolve/main/config.json from cache at .cache/huggingface/transformers/3fa0446657dd3714a950ba400a3fa72686d0f815da436514e4823a973ef23e20.7a0cb161a6d34c3881891b70d4fa06557175ac7b704a19bf0100fb9c21af9286
Model config PegasusConfig {
  "_name_or_path": "google/pegasus-large",
  "activation_dropout": 0.1,
  "activation_function": "relu",
  "add_bias_logits": false,
  "add_final_layer_norm": true,
  "architectures": [
    "PegasusForConditionalGeneration"
  ],
  "attention_dropout": 0.1,
  "bos_token_id": 0,
  "classif_dropout": 0.0,
  "classifier_dropout": 0.0,
  "d_model": 1024,
  "decoder_attention_heads": 16,
  "decoder_ffn_dim": 4096,
  "decoder_layerdrop": 0.0,
  "decoder_layers": 16,
  "decoder_start_token_id": 0,
  "dropout": 0.1,
  "encoder_attention_heads": 16,
  "encoder_ffn_dim": 4096,
  "encoder_layerdrop": 0.0,
  "encoder_layers": 16,
  "eos_token_id": 1,
  "extra_pos_embeddings": 1,
  "force_bos_token_to_be_generated": false,
  "forced_eos_token_id": 1,
  "gradient_checkpointing": false,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2"
  },
  "init_std": 0.02,
  "is_encoder_decoder": true,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2
  },
  "length_penalty": 0.8,
  "max_length": 256,
  "max_position_embeddings": 1024,
  "model_type": "pegasus",
  "normalize_before": true,
  "normalize_embedding": false,
  "num_beams": 8,
  "num_hidden_layers": 16,
  "pad_token_id": 0,
  "scale_embedding": true,
  "static_position_embeddings": true,
  "task_specific_params": {
    "summarization_aeslc": {
      "length_penalty": 0.6,
      "max_length": 32,
      "max_position_embeddings": 512
    },
    "summarization_arxiv": {
      "length_penalty": 0.8,
      "max_length": 256,
      "max_position_embeddings": 1024
    },
    "summarization_big_patent": {
      "length_penalty": 0.7,
      "max_length": 256,
      "max_position_embeddings": 1024
    },
    "summarization_billsum": {
      "length_penalty": 0.6,
      "max_length": 256,
      "max_position_embeddings": 1024
    },
    "summarization_cnn_dailymail": {
      "length_penalty": 0.8,
      "max_length": 128,
      "max_position_embeddings": 1024
    },
    "summarization_gigaword": {
      "length_penalty": 0.6,
      "max_length": 32,
      "max_position_embeddings": 128
    },
    "summarization_large": {
      "length_penalty": 0.8,
      "max_length": 256,
      "max_position_embeddings": 1024
    },
    "summarization_multi_news": {
      "length_penalty": 0.8,
      "max_length": 256,
      "max_position_embeddings": 1024
    },
    "summarization_newsroom": {
      "length_penalty": 0.8,
      "max_length": 128,
      "max_position_embeddings": 512
    },
    "summarization_pubmed": {
      "length_penalty": 0.8,
      "max_length": 256,
      "max_position_embeddings": 1024
    },
    "summarization_reddit_tifu": {
      "length_penalty": 0.6,
      "max_length": 128,
      "max_position_embeddings": 512
    },
    "summarization_wikihow": {
      "length_penalty": 0.6,
      "max_length": 256,
      "max_position_embeddings": 512
    },
    "summarization_xsum": {
      "length_penalty": 0.8,
      "max_length": 64,
      "max_position_embeddings": 512
    }
  },
  "transformers_version": "4.5.0.dev0",
  "use_cache": true,
  "vocab_size": 96103
}

loading file https://huggingface.co/google/pegasus-large/resolve/main/spiece.model from cache at .cache/huggingface/transformers/66f187d645734a6204f3fd24593fbf0d9e36b528dd85b3adae9a566b17b4768f.1acf68c74589da6c7fa3548093824dfc450a54637f4356929bbfea7e294a68f8
loading file https://huggingface.co/google/pegasus-large/resolve/main/tokenizer.json from cache at None
loading file https://huggingface.co/google/pegasus-large/resolve/main/added_tokens.json from cache at None
loading file https://huggingface.co/google/pegasus-large/resolve/main/special_tokens_map.json from cache at .cache/huggingface/transformers/fbf9c7cf2d49b24712b53a2760e7c62a2acecd1496908822df00b8ec2683ca6d.294ebaa4cd17bb284635004c92d2c4d522ec488c828dcce0c2471b6f28e3fe82
loading file https://huggingface.co/google/pegasus-large/resolve/main/tokenizer_config.json from cache at .cache/huggingface/transformers/74256fafbb3cb536e351e6731914d42f732e77d33e537b6c19fb72f4b74f50ea.43f396f0ee3b974f9128267d49f69a26b11f3ed290851ac5788a549cc2979671
loading weights file https://huggingface.co/google/pegasus-large/resolve/main/pytorch_model.bin from cache at .cache/huggingface/transformers/ef3a8274e003ba4d3ae63f2728378e73affec0029e797c0bbb80be8856130c4f.a99cb24bd92c7087e95d96a1c3eb660b51e498705f8bd068a58c69c20616f514
All model checkpoint weights were used when initializing PegasusForConditionalGeneration.

All the weights of PegasusForConditionalGeneration were initialized from the model checkpoint at google/pegasus-large.
If your task is similar to the task the model of the checkpoint was trained on, you can already use PegasusForConditionalGeneration for predictions without further training.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.43ba/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  7.76ba/s]
***** Running training *****
  Num examples = 100
  Num Epochs = 3
  Instantaneous batch size per device = 1
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 39
  0%|                                                                                                                                        | 0/39 [00:00<?, ?it/s]anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/parallel/_functions.py:65: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
  3%|███▎                                                                                                                            | 1/39 [00:32<20:45, 32.79s/it]Traceback (most recent call last):
  File "run_summarization.py", line 591, in <module>
    main()
  File "run_summarization.py", line 529, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/trainer.py", line 1120, in train
    tr_loss += self.training_step(model, inputs)
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/trainer.py", line 1524, in training_step
    loss = self.compute_loss(model, inputs)
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/trainer.py", line 1556, in compute_loss
    outputs = model(**inputs)
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 168, in forward
    return self.gather(outputs, self.output_device)
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 180, in gather
    return gather(outputs, output_device, dim=self.dim)
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 76, in gather
    res = gather_map(outputs)
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 70, in gather_map
    for k in out))
  File "<string>", line 11, in __init__
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/file_utils.py", line 1589, in __post_init__
    for element in iterator:
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 70, in <genexpr>
    for k in out))
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 71, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 71, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map
    return Gather.apply(target_device, dim, *outputs)
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 72, in forward
    return comm.gather(inputs, ctx.dim, ctx.target_device)
  File "anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/parallel/comm.py", line 235, in gather
    return torch._C._gather(tensors, dim, destination)
RuntimeError: CUDA out of memory. Tried to allocate 32.00 MiB (GPU 0; 15.78 GiB total capacity; 14.02 GiB already allocated; 21.75 MiB free; 14.16 GiB reserved in total by PyTorch)
  3%|███▎                                                                                                                            | 1/39 [00:33<21:15, 33.57s/it]
```
3. Output of GPU - config (`nvidia-smi`):

```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:17.0 Off |                    0 |
| N/A   45C    P0    45W / 300W |      3MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:00:18.0 Off |                    0 |
| N/A   42C    P0    43W / 300W |      3MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000000:00:19.0 Off |                    0 |
| N/A   42C    P0    45W / 300W |      3MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 00000000:00:1A.0 Off |                    0 |
| N/A   48C    P0    47W / 300W |      3MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-SXM2...  On   | 00000000:00:1B.0 Off |                    0 |
| N/A   45C    P0    44W / 300W |      3MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-SXM2...  On   | 00000000:00:1C.0 Off |                    0 |
| N/A   44C    P0    44W / 300W |      3MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-SXM2...  On   | 00000000:00:1D.0 Off |                    0 |
| N/A   41C    P0    44W / 300W |      3MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-SXM2...  On   | 00000000:00:1E.0 Off |                    0 |
| N/A   44C    P0    43W / 300W |      3MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
```


## Expected behavior

I expect pegasus fine tuning to succeed and produce a fine tuned model with its performance summary (rouge score).
FYI: The gist provided [here](https://gist.github.com/jiahao87/50cec29725824da7ff6dd9314b53c4b3) works.


Comments:
@patil-suraj : Any thoughts or suggestions? 

Hi @karrtikiyer 

`pegasus-large ` is quite a bid model (for a 16GB GPU), so I won't be surprised if it OOMs. I would really recommend you to try using `deepspeed` or `fairscale`, which will definitely help you to fine-tune this model more efficiently.  To give you an idea, we were able to fit 2.7B model on a single V100 using `deepspeed`. https://github.com/Xirider/finetune-gpt2xl

Here are the [docs](https://huggingface.co/transformers/main_classes/trainer.html#trainer-integrations)  for how to use `deepspeed` with `Trainer`. 

Also, I think the gist works because it freezes the encoder.

Thanks a lot @patil-suraj , will take a look at deepspeed. Really appreciate it.
It would be good if we can add in the readme file or appropriate place, which could help people like me to understand for which models we can make use of `run_summarization.py` out of box, or the cases where deepspeed or fairscale might be needed.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.