MemoryError when computing metrics on Wav2Vec2
Author: GaetanBaert
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform: Linux-5.8.0-48-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
- Wav2Vec2: @patrickvonplaten
- trainer: @sgugger
Model hub:

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)



## Information

Model I am using (Bert, XLNet ...): Wav2Vec2

The problem arises when using:
* [x] the official example scripts: https://huggingface.co/blog/fine-tune-xlsr-wav2vec2
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below) Adapting https://huggingface.co/blog/fine-tune-xlsr-wav2vec2 to French Commonvoice

## StackTrace

```
  File "train.py", line 108, in <module>
    trainer.train()
  File "/home/gaetan/anaconda3/envs/pytorch/lib/python3.8/site-packages/transformers/trainer.py", line 1105, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/home/gaetan/anaconda3/envs/pytorch/lib/python3.8/site-packages/transformers/trainer.py", line 1198, in _maybe_log_save_evaluate
    metrics = self.evaluate()
  File "/home/gaetan/anaconda3/envs/pytorch/lib/python3.8/site-packages/transformers/trainer.py", line 1667, in evaluate
    output = self.prediction_loop(
  File "/home/gaetan/anaconda3/envs/pytorch/lib/python3.8/site-packages/transformers/trainer.py", line 1840, in prediction_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=preds, label_ids=label_ids))
  File "train.py", line 94, in compute_metrics
    wer = wer_metric.compute(predictions=pred_str, references=label_str)
  File "/home/gaetan/anaconda3/envs/pytorch/lib/python3.8/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/media/gaetan/DATA/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/gaetan/anaconda3/envs/pytorch/lib/python3.8/site-packages/jiwer/measures.py", line 80, in wer
    measures = compute_measures(
  File "/home/gaetan/anaconda3/envs/pytorch/lib/python3.8/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/gaetan/anaconda3/envs/pytorch/lib/python3.8/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError
  1%|█                                                                              | 2000/147540 [2:19:05<168:41:25,  4.17s/it]

```


## Description

I'm quite surprised to get this error, when I succeeded the complete first evaluation loop (it was at the second one). 

Maybe some elements are stored in memory (I have 32GB RAM) without needing it ?
I put the option
```
eval_accumulation_steps=20
```
in my TrainingArguments because else I got CUDA Memory Errors, this is the only relevant modification I made from the example code. 

Comments:
To properly compute the metrics, the evaluation loop stores all the prediction in a NumPy array. It looks like your evaluation dataset is very large, so you should use evaluate/predict on smaller subsets then save those predictions to disk.

Thank you. I think I will make my own evaluation loop to compute metrics regularly, keep the word error rate and  the word number  (to compute average) and store only those metrics instead of the predictions, in order to save memory. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.