auto_find_batch_size for OOM during evaluation
Author: maximilianmordig
Pull request: no
Labels: name: Feature request

name: bug

### System Info

When running training and evaluation (`_inner_training_loop` in the HF `Trainer`), the auto-find-batch size tries to reduce the training batch size, even when the OOM happens during evaluation and not the training. It should distinguish whether the error comes from training or evaluation (`_maybe_log_save_evaluate`) and reduce the evaluation batch size if needed.
Also, it would be nice to report the finally chosen batch size (maybe this is already done).

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Choose a very large evaluation batch size, and a small training batch size.

### Expected behavior

Should reduce eval batch size, not just the training batch size

Comments:
Thanks for the issue @maximilianmordig! Pinging @SunMarc and @muellerzr on the `Trainer`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @maximilianmordig, `auto_find_batch` is only for the training loop. You can get the final `batch_size` by setting your logger to debug level as we have in the code the following:  `logger.debug(f"Currently training with a batch size of: {self._train_batch_size}")`
Also, are saying that it is decreasing the `batch_size` of the training even though we don't oom ? Could you share a minimal reproducer as this shouldn't happen ? 

cc @muellerzr would it make sense to extend this feature to evaluate ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Yes, it is decreasing the train batch size even though the OOM happens during eval, not during training.

Here is my temporary workaround which I have been using in the past month:
```
from transformers.trainer_utils import find_executable_batch_size
import logging

class EvaluateAutoFindBatchSizeMixin:
    def evaluate(self, *args, **kwargs):
        print(f"Auto-finding batch size for evaluate")
        # can't set self.args.eval_batch_size, it is computed from self.args.per_device_eval_batch_size
        super_evaluate = super().evaluate  # super cannot be called inside lambda

        # assert not self.args.dataloader_persistent_workers, "dataloader_persistent_workers not supported because eval batch size does not change"
        def new_evaluate(batch_size):
            old_eval_batch_size = self.args.eval_batch_size
            self.args.per_device_eval_batch_size = batch_size
            if self.args.eval_batch_size != old_eval_batch_size:
                if self.args.dataloader_persistent_workers and hasattr(
                    self, "_eval_dataloaders"
                ):
                    del self._eval_dataloaders  # to shutdown the workers
                logging.info(
                    f"Setting eval_batch_size to {self.args.eval_batch_size} (per device: {batch_size}) from old {old_eval_batch_size}"
                )
            return super_evaluate(*args, **kwargs)
            # don't reset at end, but keep batch size for the next time evaluate is called

        return find_executable_batch_size(
            new_evaluate,
            self.args.per_device_eval_batch_size,
            auto_find_batch_size=self.args.auto_find_batch_size,
        )()

from transformers import Trainer
class CustomTrainer(Trainer, EvaluateAutoFindBatchSizeMixin):
    pass
```
I am not sure if persistent workers are compatible with changing batch size.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

gentle ping @muellerzr @sunmarc - I think the stalebot picked up a bunch of issues at the same time, so sorry I keep pinging you both!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Labelling as feature request. If you are willing, would you like to open a PR to fix this @maximilianmordig  ?