Add Ray's scope to training arguments
Author: BramVanroy
Pull request: yes

# What does this PR do?

As discussed in https://github.com/huggingface/transformers/issues/16683, it often happens that you want more control over gridsearch. By default, Ray will run different trials with different hyperparameters, and then select the best one based on the loss (or other metric) of each trial's _final_ checkpoint. That's not always what you want however (when some trials converge faster than others and therefore overfit in the same number of steps). Luckily Ray allows [other options](https://docs.ray.io/en/latest/tune/api_docs/analysis.html#ray.tune.ExperimentAnalysis.get_best_trial). With this PR, the user gets a little bit more control over how to use Ray's search.

<!-- Remove if not applicable -->

Fixes #16683 


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. **Discussed but never received an official reply. Got some community responses of others who would like to see this change, however.**
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).


## Who can review?

@richardliaw, @amogkam


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Awesome stuff :) @Yard1 could you help shepherd this PR?

@BramVanroy Hey this looks great! Looks like you only need to run black to reformat the files.

@richardliaw @Yard1 I can't manage to satisfy the quality checks (`doc-builder`). The issue being the very long Ray URL. Any thoughts how I should deal with this? 

Hey @BramVanroy, running `make fixup` at the root of your clone should fix it. It should apply a fix such as the following:

```diff
diff --git a/src/transformers/training_args.py b/src/transformers/training_args.py
index 23f8cd752..e54227b46 100644
--- a/src/transformers/training_args.py
+++ b/src/transformers/training_args.py
@@ -459,10 +459,9 @@ class TrainingArguments:
         ray_scope (`str`, *optional*, defaults to `"last"`):
             The scope to use when doing hyperparameter search with Ray. By default, `"last"` will be used. Ray will
             then use the last checkpoint of all trials, compare those, and select the best one. However, other options
-            are also available. See the
-            [Ray documentation](
-            https://docs.ray.io/en/latest/tune/api_docs/analysis.html#ray.tune.ExperimentAnalysis.get_best_trial)
-            for more options.
+            are also available. See the [Ray documentation](
+            https://docs.ray.io/en/latest/tune/api_docs/analysis.html#ray.tune.ExperimentAnalysis.get_best_trial) for
+            more options.
     """
 
     output_dir: str = field(
```

In general, you can break up very long links with `\`, eg.
```
        ray_scope (`str`, *optional*, defaults to `"last"`):
            The scope to use when doing hyperparameter search with Ray. By default, `"last"` will be used. Ray will
            then use the last checkpoint of all trials, compare those, and select the best one. However, other options
            are also available. See the
            [Ray documentation](
            https://docs.ray.io/en/latest/tune/api_docs/analysis.html\
#ray.tune.ExperimentAnalysis.get_best_trial)
            for more options.
```

cc @sgugger for the additional argument to the trainings arguments