Bug on Gather all remaining tensors and put them back on the CPU
Author: jinmang2
Pull request: no

### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger @muellerzr @ArthurZucker 

### Information

- [X] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

### Colab Link
- https://colab.research.google.com/drive/1afT8O5OrUaTaZ07xvi_3AMW07P2nnRuC?usp=sharing

### Expected behavior

### What is problem?
In the trainer's `evaluation_loop`, the huggingface trainer collects all tensors and puts them into the `compute_metrics` method all at once. If the tensor is too large, a CUDA OOM error occurs, so it's prevented by sending it to the cpu in advance (using `nested_numpify` method) in the middle with `eval_accumulation_step`.
For samples smaller than `eval_accumulation_step`, metrics were calculated after additional summing using the code below.

https://github.com/huggingface/transformers/blob/66fd3a8d626a32989f4569260db32785c6cbf42a/src/transformers/trainer.py#L3304-L3318

However, since the code changes as below in PR #24028, the problem as occurred.

https://github.com/huggingface/transformers/blob/ebd94b0f6f215f6bc0f70e61eba075eb9196f9ef/src/transformers/trainer.py#L3184-L3192

The code above doesn't merge the remaining tensors into the final container, it just allocates them. In fact, in the example code I ran, even though `len(eval_dataset)` was `7224`, with `per_device_eval_batch_size=16`, `eval_accumulation_steps=100`, gpu-cpu communication was performed 4 times, and only `824` eval samples remained. 

Please check it out and hope the PR will be corrected. Thank you!

Comments:
@jinmang2 can you provide the example code you ran? Would be an excellent way for  us to write a test-case around it :)

@muellerzr sure!

### dialogue state tracking results (my own code)
- colab link: https://colab.research.google.com/drive/1afT8O5OrUaTaZ07xvi_3AMW07P2nnRuC?usp=sharing
- code link: https://github.com/jinmang2/KLUE-DST/blob/main/run.py
- results

```python
...
eval_results = trainer.evaluation_loop(
    trainer.get_eval_dataloader(),
    description="Evaluation",
    prediction_loss_only=False,
    ignore_keys=None,
    metric_key_prefix="eval",
)
len(trainer.eval_dataset), eval_results.predictions[0].shape
```
```
(7224, (824, 9, 71))
```
- expected shape: `(7224, 9, 71)`

### glue mnli results (huggingface's example code)
- colab link: https://colab.research.google.com/drive/1Yfoh4-Pl5LqGUWBZZqbc3OGN1R3x3O_w?usp=sharing
- code link: https://github.com/huggingface/transformers/blob/ba695c1efd55091e394eb59c90fb33ac3f9f0d41/examples/pytorch/text-classification/run_glue.py
- results

```python
...
eval_results = trainer.evaluation_loop(
    trainer.get_eval_dataloader(),
    description="Evaluation",
    prediction_loss_only=False,
    ignore_keys=None,
    metric_key_prefix="eval",
)
# The total number of samples in the eval example is 9815.
# However, it can be seen that only 3 samples of prediction used for evaluation remain.
len(trainer.eval_dataset), eval_results.predictions[0].shape
```
```
(9815, (3,))
```
- expected shape: `(9815,)`

Since the `evaluate` method does not know how many eval samples have been evaluated (internally, only necessary values ​​are loaded into the metrics dictionary), the `evaluation_loop` method directly receives `eval_results` and checks the eval samples.

Thanks @jinmang2, this indeed was a big from the integration and the original logic should have been maintained. A PR will be opened shortly with the solution (and also solves a failing test!) thanks again for your solution and thorough analysis

Thanks for fixing it! :-)