Issues occuring during parallel evaluation (using Trainer.evaluate)
Author: psychocosine
Pull request: no
Labels: name: trainer

### System Info

- `transformers` version: 4.40.0
- Platform: Linux-5.15.0-105-generic-x86_64-with-glibc2.31
- Python version: 3.9.19
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.3
- Accelerate version: 0.29.3
- Accelerate config:    not found
- PyTorch version (GPU?): 1.13.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@muellerzr 

### Information

- [x] The official example scripts
- [X] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The length of the ```eval_preds``` parameter received in the ```compute_metrics``` function is different from the original length in ```eval_dataset```. 
```python
    def compute_metrics(eval_preds):
        preds, labels = eval_preds
        # In case the model returns more than the prediction logits
        if isinstance(preds, tuple):
            preds = preds[0]
        assert preds.shape[-1] == training_args.max_length
        assert preds.shape[0] == len(tokenized_datasets[-1]) #### assertion error preds.shape[0]=1024 ,len(tokenized_datasets[-1])=1012
```

```python
trainer = Trainer(
        model,
        training_args,
        train_dataset=tokenized_datasets[0].shuffle(seed=42).select(range(int(1e6))),
        eval_dataset={data_args.task_name: tokenized_datasets[-1]},
        data_collator=data_collator,
        tokenizer=tokenizer,
        compute_metrics=compute_metrics,
        callbacks=[LoggerCallback, DenserEvalCallback]
    )
```
My training args lists below :
```python
n_gpus=2
per_device_train_batch_size=8
per_device_eval_batch_size=8
gradient_accumulation_steps=3
len(preds)=1024
len(tokenized_datasets[-1])=1012
```


### Expected behavior

Everything works fine when using single gpu but not gpus
I started my script by calling ```accelerate launch scripy.py```

Comments:
Gentle ping @muellerzr 

Another ping @muellerzr @SunMarc 

Hey @psychocosine, sorry for the huge delay. This doesn't seems correct. Can you check if you still have the issue with the latest transformers. Could you also share a minimal reproducer ? It looks like `gather_for_metrics` wasn't called or maybe there an issue with the function. 

The reason why there are more evals preds when using distributed evaluation is that we are adding a few extra samples in the eval dataset in order to have the same number of data in both gpus, so that it is easier to gather them ! See more information in this accelerate [doc](https://huggingface.co/docs/accelerate/en/quicktour#distributed-evaluation). 

In your example, len(tokenized_datasets[-1])=1012 and per_device_eval_batch_size=8. So it means that we have 1012/2=506 sample for each gpus but 506/8= 63.25 batch which is not a round number. So we need 64 batch (506 samples + 6 extra samples) for each gpus. After gathering, the total number of sample is 512 * 2 = 1024. Of course, we will drop the prediction for the extra sample by calling `gather_for_metrics`. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.