OOM Issue when evaluating with Trainer
Author: Skyy93
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.1.0
- Platform: Ubuntu 20.04
- Python version: Python 3.8.2
- PyTorch version (GPU?): 1.5.1
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
 
 Trainer: @sgugger 

## Information

Model I am using BartForConditionalGeneration:

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

Using the normal Trainer with trainer.evaluate() on the BillSum Dataset

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

BillSum Dataset

## To reproduce

Steps to reproduce the behavior:

1. Load Billsum
2. FineTune Bart on Billsum 
3. try to evaluate with trainer.evaluate()

`
Traceback (most recent call last):
  File "main.py", line 211, in <module>
    main()

  File "main.py", line 182, in main
    trainer.evaluate()

  File "/home/mypc/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1157, in evaluate
    output = self.prediction_loop(eval_dataloader, description="Evaluation")
  File "/home/mypc/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1244, in prediction_loop
    preds = logits if preds is None else torch.cat((preds, logits), dim=0)
RuntimeError: CUDA out of memory. Tried to allocate 2.58 GiB (GPU 0; 7.79 GiB total capacity; 2.72 GiB already allocated; 1.94 GiB free; 4.88 GiB reserved in total by PyTorch)
`

## Expected behavior

The Tensors for the logits are getting really big (x, 1024, 52000) and will be concatenated every prediction step in the trainer.py in line 1244:
`             
if logits is not None:
    preds = logits if preds is None else torch.cat((preds, logits), dim=0)
` 
One possible solution is to argmax them while iterating through the prediction loop

Comments:
We know there is a limitation on the size of dataset for evaluation/prediction because every tensors are concatenated together (this is necessary for distributed training). We plan to work on it using the work done in the `datasets` library to have the predictions cached in temporary files as the prediction loop go. However this is not going to be ready before a few weeks, so in the meantime, I encourage you to feed your dataset to predictions by small slices.

Okay, thank you. 

Currently i'm only debugging on my machine so the:
` if logits is not None: preds = logits.argmax(-1) if preds is None else torch.cat((preds, logits.argmax(-1)), dim=0)`
-hack does it for me. But yeah as soon as I train on multiple machines I will feed it in small slices.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> We know there is a limitation on the size of dataset for evaluation/prediction because every tensors are concatenated together (this is necessary for distributed training). We plan to work on it using the work done in the `datasets` library to have the predictions cached in temporary files as the prediction loop go. However this is not going to be ready before a few weeks, so in the meantime, I encourage you to feed your dataset to predictions by small slices.

Hi, has this problem been solved :)