The Trainer uses all available GPU devices when training but only one when evaluating.
Author: seanswyi
Pull request: no

### System Info

- `transformers` version: 4.37.2
- Platform: Linux-4.15.0-76-generic-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.26.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.0+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes, but not explicitly using any code and relying on Trainer.
- Using distributed or parallel set-up in script?: Same as above.

### Who can help?

@muellerzr @pacman100 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I'm using a custom dataset for NER and am doing the following:

    from transformers import (
        AutoTokenizer,
        DataCollatorForTokenClassification,
        Trainer,
        TrainingArguments,
    )

    data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)

    trainer = Trainer(
        model=ner_model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=train_dataset,
        eval_dataset=valid_dataset,
        tokenizer=tokenizer,
    )
    trainer.train()


### Expected behavior

I currently have 4 GPU devices available and have not explicitly set `CUDA_VISIBLE_DEVICS`. I've used Accelerate before and not setting this environment variable doesn't seem to be a problem since I believe the default behavior is to search for all available devices and use them.

This happens fine during training but during evaluation I'm not sure why only one GPU is being used.

Comments:
I'll bump this I have the same issue

I haven't had the time to actually look into this properly, but my intuition is that perhaps this is a design choice since evaluating across multiple devices may result in erroneous results (e.g., errors when gathering). That's why I usually use all devices for evaluating on the validation set but only use one device for evaluating on the test set. 

But it only uses one device when on the validation set as well. How do you get it to differentiate between test and validation in the trainer? How do you know there'd be errors when gathering?

I don't think that you can do that with the current HuggingFace API since it only appears to be using a `train_dataset` and `eval_dataset`. I believe that if you want to use a third dataset (i.e., a test set) then you'd have to create a separate Trainer object because you would need to prepare your data with Accelerate again.

> How do you know there'd be errors when gathering?

I don't think you can know ahead of time 100%, but I have experienced cases where the gathering has resulted in erroneous values for the number of samples (e.g., the `support` of scikit-learn's classification report showing a different number, etc.).

I'm personally planning on taking a better look at this some time this week when I have more time outside of work.

I fixed the problem by making `eval_accumulation_steps` equal to the number of GPU devices.

Glad you fixed it but that still seems a bit counterintuitive to me. I don't recall seeing anything about that in the documentation either. I'll close this for now and reopen it later when something comes up.

@seanswyi It's OK to leave this open if you think there's still changes needed in the documentation for clarification. 

@amyeroberts This isn't related to this issue itself, but do you know if there is a particular reason behind the design choice to only use a train and eval dataset rather than allowing the option to include a test set? I'm wondering if including that option would go against some sort of design principle.

@seanswyi I don't know tbh. My guess would be for cleanliness and to prevent peeking behaviour. Evaluating on a test set is separate from the training process, which we can use both training and validation metrics to track the model performance and make decisions on e.g. hyperparam sweeps. So it might not make sense to be part of a "trainer".

cc @pacman100 @muellerzr to confirm and address the docstrings. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.