OOM issue with prediction
Author: XinnuoXu
Pull request: no

Hi! I fine-tuned the bart model on XSum (both training and validation are fine). However, the OOM appeared during the prediction using the same machine.  @patrickvonplaten @patil-suraj Here is my code:

```
python3 run_summarization.py \
        --output_dir ./tmp/xsum-test/ \
        --overwrite_output_dir \
        --text_column text \
        --summary_column summary \
        --per_device_eval_batch_size 1 \
        --do_predict \
        --model_name_or_path ./tmp/xsum-summarization/checkpoint-15000 \
        --max_source_length=512 \
        --max_target_length=128 \
        --val_max_target_length=60 \
        --test_path data/multi \
        --num_beams 6 \
```

The error is:

```
***** Running Prediction *****
  Num examples = 11334
  Batch size = 1
  4%|▍         | 465/11334 [00:54<42:38,  4.25it/s]Traceback (most recent call last):
  File "run_summarization.py", line 587, in <module>
    main()
  File "run_summarization.py", line 559, in main
    num_beams=data_args.num_beams,
  File "/lustre/home/ec156/xx6/transformers/src/transformers/trainer_seq2seq.py", line 121, in predict
    return super().predict(test_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "/lustre/home/ec156/xx6/transformers/src/transformers/trainer.py", line 1824, in predict
    test_dataloader, description="Prediction", ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix
  File "/lustre/home/ec156/xx6/transformers/src/transformers/trainer.py", line 1900, in prediction_loop
    preds_host = logits if preds_host is None else nested_concat(preds_host, logits, padding_index=-100)
  File "/lustre/home/ec156/xx6/transformers/src/transformers/trainer_pt_utils.py", line 96, in nested_concat
    return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
  File "/lustre/home/ec156/xx6/transformers/src/transformers/trainer_pt_utils.py", line 96, in <genexpr>
    return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
  File "/lustre/home/ec156/xx6/transformers/src/transformers/trainer_pt_utils.py", line 98, in nested_concat
    return torch_pad_and_concatenate(tensors, new_tensors, padding_index=padding_index)
  File "/lustre/home/ec156/xx6/transformers/src/transformers/trainer_pt_utils.py", line 66, in torch_pad_and_concatenate
    result = tensor1.new_full(new_shape, padding_index)
RuntimeError: CUDA out of memory. Tried to allocate 932.00 MiB (GPU 0; 15.78 GiB total capacity; 12.89 GiB already allocated; 913.69 MiB free; 13.79 GiB reserved in total by PyTorch)
  4%|▍         | 465/11334 [00:55<21:27,  8.44it/s]srun: error: r2i4n0: task 0: Exited with exit code 1
```

Comments:
Hi @XinnuoXu 

you should pass the `--predict_with_generate` arg for summarization evaluation, this will use the `generate` method to generate the summaries. 

I think one possible reason for this issue is that when `predict_with_generate` is not passed the final hidden_states from the model are used as predictions which are of shape `[bs, seq_len, vocab_size]`, which is quite large, hence OOM.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.