Add the auto_find_batch_size capability from Accelerate into Trainer
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR introduces the `find_executable_batch_size` decorator into `Trainer`, so the training loop is repeated if a CUDA OOM is reached, lowering the batch size. 

The API looks as so:

```python
trainer = Trainer()
trainer.train(auto_find_batch_size=True)
```
By default it is False, and requires `Accelerate` be installed to use. 

Fixes # (issue)

Partially solves https://github.com/huggingface/transformers/issues/16987

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 

Comments:
@stas00 I'm getting a test failure on the metrics:

```python
 tests/trainer/test_trainer.py:1426: in check_mem_metrics
     metrics = trainer.train().metrics
 src/transformers/trainer.py:1215: in train
     ignore_keys_for_eval=ignore_keys_for_eval,
 src/transformers/trainer.py:1571: in _inner_training_loop
     self._memory_tracker.stop_and_update_metrics(metrics)
 src/transformers/trainer_utils.py:536: in stop_and_update_metrics
     stage = self.derive_stage()
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 
 self = <transformers.trainer_utils.TrainerMemoryTracker object at 0x7f929f787e10>
 
     def derive_stage(self):
         """derives the stage/caller name automatically"""
         caller = inspect.currentframe().f_back.f_back.f_code.co_name
         if caller in self.stages:
             return self.stages[caller]
         else:
             raise ValueError(
 >               f"was called from {caller}, but only expect to be called from one of {self.stages.keys()}"
             )
 E           ValueError: was called from _inner_training_loop, but only expect to be called from one of dict_keys(['__init__', 'train', 'evaluate', 'predict'])
```

Any advice on how to approach a solution?

_The documentation is not available anymore as the PR was closed or merged._

This will overcome the problem:

```
diff --git a/src/transformers/trainer_utils.py b/src/transformers/trainer_utils.py
index 22b44a2f0..d4c523249 100644
--- a/src/transformers/trainer_utils.py
+++ b/src/transformers/trainer_utils.py
@@ -356,6 +356,7 @@ class TrainerMemoryTracker:
     stages = {
         "__init__": "init",
         "train": "train",
+        "_inner_training_loop": "train",
         "evaluate": "eval",
         "predict": "test",
     }
```


Please make sure all tests pass after resolving conflicts and before merging!

Any chance similar functionality could be supported for inference? 🙏