Some feature requests for the Trainer
Author: NielsRogge
Pull request: no

### Feature request

2 feature requests for the HuggingFace Trainer:

* seems like the Trainer is currently using `huggingface_hub.Repository` when pushing a model to the hub. Would be great to update this to leverage the new HTTP methods, as currently I'm getting errors like the following:
```
OSError                                   Traceback (most recent call last)
[<ipython-input-37-be354f2ba166>](https://localhost:8080/#) in <module>
----> 1 trainer.push_to_hub("nielsr/layoutxlm-xfund-fr-relation-extraction")

3 frames
[/usr/local/lib/python3.7/dist-packages/transformers/trainer.py](https://localhost:8080/#) in push_to_hub(self, commit_message, blocking, **kwargs)
   3373         # it might fail.
   3374         if not hasattr(self, "repo"):
-> 3375             self.init_git_repo()
   3376 
   3377         if self.args.should_save:

[/usr/local/lib/python3.7/dist-packages/transformers/trainer.py](https://localhost:8080/#) in init_git_repo(self, at_init)
   3255                 clone_from=repo_name,
   3256                 use_auth_token=use_auth_token,
-> 3257                 private=self.args.hub_private_repo,
   3258             )
   3259         except EnvironmentError:

[/usr/local/lib/python3.7/dist-packages/huggingface_hub/repository.py](https://localhost:8080/#) in __init__(self, local_dir, clone_from, repo_type, use_auth_token, git_user, git_email, revision, private, skip_lfs_files, client)
    496 
    497         if clone_from is not None:
--> 498             self.clone_from(repo_url=clone_from)
    499         else:
    500             if is_git_repo(self.local_dir):

[/usr/local/lib/python3.7/dist-packages/huggingface_hub/repository.py](https://localhost:8080/#) in clone_from(self, repo_url, use_auth_token)
    725                 if not in_repository:
    726                     raise EnvironmentError(
--> 727                         "Tried to clone a repository in a non-empty folder that isn't a"
    728                         " git repository. If you really want to do this, do it"
    729                         " manually:\ngit init && git remote add origin && git pull"

OSError: Tried to clone a repository in a non-empty folder that isn't a git repository. If you really want to do this, do it manually:
git init && git remote add origin && git pull origin main
 or clone repo to a new folder and move your existing files there afterwards.
````
* would be great to have an argument `push_to_hub_frequency`, to indicate at which steps to push the model to the hub (seems like it's pushing to the hub every epoch at the moment by default).

### Motivation

Improving the push to hub functionalities of the Trainer.

### Your contribution

I hope @sgugger has the bandwidth to work on this :D

Comments:
For 1, just overwrite the output dir with `--overwrite_output_dir` to have a clean output dir for the beginning of training (when resuming there shouldn't be any error since the repo will be synced with the local folder). If we stop leveraging `Repository`, we lose the async pushes so basically training will be interrupted each time there is a push, until the push is finished.

For 2, pushes are synced with saves, so change your `save_strategy` to `"steps"` and set the `save_steps` to the value of your liking.

Thanks for clarifying, makes sense!