a memory leak in qqp prediction using bart
Author: StevenTang1998
Pull request: no

### System Info

```shell
- `transformers` version: 4.19.0.dev0
- Platform: Linux-5.11.0-43-generic-x86_64-with-glibc2.17
- Python version: 3.8.10
- Huggingface_hub version: 0.4.0
- PyTorch version (GPU?): 1.10.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
```


### Who can help?

@sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I met the same issue #11011. If not using `--eval_accumulation_steps`, it caused CUDA out of memory. If using it, it caused out of RAM and killed by system.

I only did prediction on GLUE QQP dataset using bart without fine-tuning. Considering QQP having a large test set (300k), the prediction got slower and slower, and finally got out of memory.

This is the script to reproduce:
```
CUDA_VISIBLE_DEVICES=0 python run_glue.py   --model_name_or_path facebook/bart-large   --task_name qqp   --output_dir bart-large_qqp --eval_accumulation_steps 100 --do_predict --per_device_eval_batch_size 24
```

### Expected behavior

```shell
Prediction without out memory.
```


Comments:
There is nothing we can do to help with that, as you don't seem to have the RAM necessary to hold all predictions. The only advise I have is that you should predict on parts of the dataset and not the whole. 

This is not a bug in Transformers, removing the label.

Sorry, I don't think that. I have 512GB RAM. And I can conduct training and evaluation well, but this issue only occurs during prediction.

The training does not accumulate predictions and the evaluation uses the evaluation set which is smaller.

But I don't understand why a tensor of shape (300k,) will exceed RAM? Does trainer save intermediate hidden states during prediction?

Mmmmmm, it may be that the model is outputting more tensors than just the logits. I see it has `use_cache=True` in its config, can you try again by setting it to `False`?

OK! I try it now! Thanks!

Sorry, it doesn't work.

It takes more time and more memory with every 100 steps.

I have found how to solve it!

The `logits` in the [https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L2635](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L2635) returns two tensors of shape (batch_size, num_classes) and (batch_size, max_sequence_length, hidden_size).
And the second shape will be accumulated, and finally will become a tensor of (300k, 256, 1024), and it will be out of memory.

And if i don't accumulate it, the code can work well.

Yes, that is what I was saying earlier: the model does not return the predictions only but some hidden states. Not sure which option will deactivate the second one.

Yes, thanks for your help! And it would be nice if this could be improved.