HPO could be enabled by a HPO configuration file(yaml or json) instead of adding code explicitly in example.py
Author: sywangyi
Pull request: no

### Feature request

now, the HPO should be enabled by adding some code in example.py(like run_glue.py) to indicate the HPO backend, metric, hp space. code like
```
     def glue_hp_space(trial):
          return [
               {"bounds": {"min": 1e-6, "max": 1e-4}, "name": "learning_rate", "type": "double"},
               {
                   "categorical_values": ["16", "32", "64", "128"],
                   "name": "per_device_train_batch_size",
                   "type": "categorical",
               },
           ]
    def model_init(trial):
            return AutoModelForSequenceClassification.from_pretrained(
                model_args.model_name_or_path,
                from_tf=bool(".ckpt" in model_args.model_name_or_path),
                config=config,
                cache_dir=model_args.cache_dir,
                revision=model_args.model_revision,
                use_auth_token=True if model_args.use_auth_token else None,
            )

        # Initialize our Trainer
       trainer = Trainer(
            model=model,
            args=training_args,
            train_dataset=train_dataset,
            eval_dataset=eval_dataset,
            compute_metrics=compute_metrics,
            model_init=model_init,
            tokenizer=tokenizer,
            data_collator=data_collator,
        )
        best_trial = trainer.hyperparameter_search(
            direction="maximize", backend="sigopt", hp_space=glue_hp_space, n_trials=2
        )
```
could we add a configuration file and pass it as an argument to trainer for the easy usage of the HPO?
yaml could like 
```
HPO.yaml

name: hg_glue_optimization
backend:  sigopt
metrics:
 -name: accuracy
  strategy: optimize
  objective: maximize
parameters:
 -name: learning_rate
   bounds:
      min: 1e-6
      max: 1e-4
   type: double
 -name: per_device_train_batch_size
   categorical_values:
      - 16
      - 32
      - 64
      - 128
    type: categorical_values
trials: 10

```
training_args could be responsible for the yaml parse and convert to input space format according to the different backend (Optuna, Sigopt, Wandb...)


### Motivation

I am always frustrated when I need to modify the example code to enable HPO, and change it according to the different HPO backend

### Your contribution

I could help submit PR after all are aligned on this point 

Comments:
@sgugger @yao-matrix @kding1  any comment?

The examples are meant to stay simple and readable so users can change them to their need (cause as mentioned in several places, they are just examples, not production apps). That's why we set the bar at training + evaluation and they don't support hyperparameters search out of the box.

> 

Agree that examples should stay simple and clear for easily ramp up. @sgugger , our question is actually "is it needed to unify HF's HPO configuration and run across different backends(Optuna, SigOpt etc.) ?",  by supplying an unified configure interface(one example is the yaml style @sywangyi proposed). For data scientist, they can decouple their applications code from specific HPO tool; for HPO tool developer, easier for them to integrate into HF ecosystem. 

@sgugger  thanks for your comment, do you think it's necessary to apply the same yaml configuration in case the user is not quite familiar with so much different HPO backend?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.