How to get best model from hyperparameter search easily
Author: sven-h
Pull request: no

Hi,

after doing a hyperparameter search(by calling `hyperparameter_search`on the trainer object) , I asked myself how to easily get the best model out of it. Currently, I'm using Ray Tune as a backend.
Given [the code in integrations.py](https://github.com/huggingface/transformers/blob/54e55b52d4886d4c63e592310b4253e01c606285/src/transformers/integrations.py#L238) the trial id, objective and chosen hyperparameters are stored in class [BestRun](https://github.com/huggingface/transformers/blob/54e55b52d4886d4c63e592310b4253e01c606285/src/transformers/trainer_utils.py#L116) which is then returned by the hyperparameter_search function. But here the model is somehow missing or am I wrong? One option would be to retrained from the given hyperparameters but this is not possible in PBT because the perturbation is applied during hyperparameter search (and cannot be repeated).

The only thing I currently see is to load the model based on the run_id and compose the corresponding file path. 
But maybe there is an easier way to do it (or is this the expected way?). I also tried out the parameter `load_best_model_at_end=True` in [TrainingArguments](https://huggingface.co/transformers/main_classes/trainer.html#trainingarguments) which is used in [this example](https://docs.ray.io/en/master/tune/examples/pbt_transformers.html) but it does not help.
One proposal would be to add a parameter in BestRun which would contain the model or even better load it directly into the trainer (such that the `predict` and `save_model` functions also work).
Is this a reasonable feature request? In case yes, I'd be happy to create a pull request.

In other documentations they show how they extract the checkpoint directly:
- [Ray Docs](https://docs.ray.io/en/master/tune/tutorials/tune-serve-integration-mnist.html#configuring-the-search-space-and-starting-ray-tune)
```
best_trial = analysis.get_best_trial("mean_accuracy", "max", "last")
best_accuracy = best_trial.metric_analysis["mean_accuracy"]["last"]
best_trial_config = best_trial.config
best_checkpoint = best_trial.checkpoint.value
```
- [Colab example for PBT](https://colab.research.google.com/drive/1tQgAKgcKQzheoh503OzhS4N9NtfFgmjF?usp=sharing#scrollTo=TxKyvQ6WNlvG)
```
best_config = analysis.get_best_config(metric="eval_acc", mode="max")
print(best_config)
best_checkpoint = recover_checkpoint(
    analysis.get_best_trial(metric="eval_acc",
                            mode="max").checkpoint.value)
```

Best regards
Sven

Comments:
cc @sgugger 

Yes there is nothing available for that right now. I believe the to `run_hp_search` functions should save the checkpoints of the non-aborted training and at least return the location of the best checkpoint in the BestRun namedtuple, as well as load the best model fine-tuned at the end if `load_best_model_at_end=True`. If you want to tackle this, we'd love to get a PR!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.