iterative evaluation in Trainer to save memory
Author: PaulLerner
Pull request: no

# 🚀 Feature request

In Trainer.prediction_loop, compute metrics batch per batch (or allow to tweak this number) instead of gathering the whole predictions in a single array (nb: I'm aware of `eval_accumulation_steps` but it only allows to save GPU memory)

## Motivation

Running a ses2seq evaluation on a dataset of 200K examples with a vocabulary of 50K and context of 77 words, gathering all of the output amounts to an array of 2.77 TiB, which I'm not sure everyone can afford:
```
                                        Traceback (most recent call last):
  File "/mnt/beegfs/home/lerner/anaconda3/envs/transformers/bin/clip-train", line 33, in <module>
    sys.exit(load_entry_point('clip', 'console_scripts', 'clip-train')())
  File "/mnt/beegfs/home/lerner/CLIP/clip/train.py", line 196, in main
    trainer.train(**config.get("checkpoint", {}))
  File "/mnt/beegfs/home/lerner/anaconda3/envs/transformers/lib/python3.7/site-packages/transformers/trainer.py", line 983, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/mnt/beegfs/home/lerner/anaconda3/envs/transformers/lib/python3.7/site-packages/transformers/trainer.py", line 1058, in _maybe_log_save_evaluate
    metrics = self.evaluate()
  File "/mnt/beegfs/home/lerner/anaconda3/envs/transformers/lib/python3.7/site-packages/transformers/trainer_seq2seq.py", line 74, in evaluate
    return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "/mnt/beegfs/home/lerner/anaconda3/envs/transformers/lib/python3.7/site-packages/transformers/trainer.py", line 1513, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/mnt/beegfs/home/lerner/anaconda3/envs/transformers/lib/python3.7/site-packages/transformers/trainer.py", line 1644, in prediction_loop
    preds_gatherer.add_arrays(self._gather_and_numpify(preds_host, "eval_preds"))
  File "/mnt/beegfs/home/lerner/anaconda3/envs/transformers/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 330, in add_arrays
    self._storage = nested_new_like(arrays, self.total_samples, padding_index=self.padding_index)
  File "/mnt/beegfs/home/lerner/anaconda3/envs/transformers/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 238, in nested_new_like
    return np.full_like(arrays, padding_index, shape=(num_samples, *arrays.shape[1:]))
  File "<__array_function__ internals>", line 6, in full_like
  File "/mnt/beegfs/home/lerner/anaconda3/envs/transformers/lib/python3.7/site-packages/numpy/core/numeric.py", line 382, in full_like
    res = empty_like(a, dtype=dtype, order=order, subok=subok, shape=shape)
  File "<__array_function__ internals>", line 6, in empty_like
MemoryError: Unable to allocate 2.77 TiB for an array with shape (200206, 77, 49408) and data type float32
 60%|██████    | 3000/5000 [17:53<11:55,  2.80it/s]

```

### Who can help
Library:
 * trainer: @sgugger


Comments:
This would result in an inaccurate metric in most cases, as metric functions are seldom linear. I'm afraid you will have to run evaluation on smaller chunks or use your own evaluation loop.

Yes of course I thought maybe to gather a reduced version of the predictions before actually computing the metric

e.g. for sentence level accuracy, in my example, a bool array of shape `(200206, )` where the boolean value represents the accuracy of the output (i.e. `predictions == labels`). 

The actual `compute_metrics` would only have to reduce this array to a single value (using `np.mean` in my example).

You can do that using `Trainer` if your model returns that. `Trainer` is too generic to be able to guess that in this case it should gather a reduced version of the predictions (and how would it do it?). Otherwise writing the evaluation loop yourself is super easy (there is one example in [run_glue_no_trainer](https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_glue_no_trainer.py) for instance).

Ok, thanks for the advice :)