Add hyperparameter search to Trainer
Author: sgugger
Pull request: yes

This PR introduces an access point to easily use optuna or Ray Tune hyperparameter search with `Trainer`.

Here is an example of use on MRPC (requires nlp installed from source on branch cloudpickle):
```
from nlp import load_dataset, load_metric
from transformers import AutoModelForSequenceClassification, AutoTokenizer, DataCollatorWithPadding, Trainer, TrainingArguments

tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
dataset = load_dataset('glue', 'mrpc')
metric = load_metric('glue', 'mrpc')

def encode(examples):
    outputs = tokenizer(examples['sentence1'], examples['sentence2'], truncation=True)
    return outputs

encoded_dataset = dataset.map(encode, batched=True)
# Won't be necessary when this PR is merged with master since the Trainer will do it automatically
encoded_dataset.set_format(columns=['attention_mask', 'input_ids', 'token_type_ids', 'label'])

def model_init():
    return AutoModelForSequenceClassification.from_pretrained('bert-base-cased', return_dict=True)

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = predictions.argmax(axis=-1)
    return metric.compute(predictions=predictions, references=labels)

# Evaluate during training and a bit more often than the default to be able to prune bad trials early.
# Disabling tqdm is a matter of preference.
training_args = TrainingArguments("test", evaluate_during_training=True, eval_steps=500, disable_tqdm=True)
trainer = Trainer(
    args=training_args,
    data_collator=DataCollatorWithPadding(tokenizer),
    train_dataset=encoded_dataset["train"], 
    eval_dataset=encoded_dataset["validation"], 
    model_init=model_init,
    compute_metrics=compute_metrics,
)

# Defaut objective is the sum of all metrics when metrics are provided, so we have to maximize it.
trainer.hyperparameter_search(direction="maximize")
```
This will use optuna or ray Tune depending on which platform is installed (default to optuna if both are there). You can force a backend by passing `backend="ray"` (or  `backend="optuna"`).

Here is another example on a simple regression problem (will convert to test in a follow-up PR):
```
from transformers import Trainer, TrainingArguments
import numpy as np

class RegressionDataset:
    def __init__(self, a=2, b=3, length=64, seed=42):
        np.random.seed(seed)
        self.length = length
        self.x = np.random.normal(size=(length,)).astype(np.float32)
        self.y = a * self.x + b + np.random.normal(scale=0.1, size=(length,)).astype(np.float32)
    
    def __len__(self):
        return self.length
    
    def __getitem__(self, i):
        return {'input_ids': self.x[i], 'label': self.y[i]}

class RegressionModel(torch.nn.Module):
    def __init__(self, a=0, b=0):
        super().__init__()
        self.a = torch.nn.Parameter(torch.tensor(a).float())
        self.b = torch.nn.Parameter(torch.tensor(b).float())
    
    def forward(self, input_ids=None, labels=None):
        y = input_ids * self.a + self.b
        if labels is None:
            return (y,)
        loss = torch.nn.functional.mse_loss(y, labels)
        return (loss, y)

train_set = RegressionDataset()
eval_set = RegressionDataset()

model = RegressionModel()
training_args = TrainingArguments("test", evaluate_during_training=True, eval_steps=8, disable_tqdm=True)

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    true = np.abs(predictions - labels) <= 0.25
    return {'accuracy': true.astype(np.float32).mean().item()}

trainer = Trainer(args=training_args, train_dataset=train_set, eval_dataset=eval_set, model_init=lambda: RegressionModel(), compute_metrics=compute_metrics)

best_trial = trainer.hyperparameter_search(direction="maximize")
```

To customize the hyperparameter searched, pass along a `hp_space` function like this (for optuna):
```
def my_hp_space(trial):
    return {
        "learning_rate": trial.suggest_float("learning_rate", 1e-2, 1, log=True),
        "num_train_epochs": trial.suggest_int("num_train_epochs", 1, 5),
        "seed": trial.suggest_int("seed", 1, 40),
        "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [4, 8, 16, 32, 64]),
    }
best_trial = trainer.hyperparameter_search(direction="maximize", hp_space=my_hp_space)
```

To customize the objective, pass along a `compute_objective` function.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6576?src=pr&el=h1) Report
> Merging [#6576](https://codecov.io/gh/huggingface/transformers/pull/6576?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d9cbc0350deaa7e146a8c8dbb6ad4dc9bd6afc4f?el=desc) will **decrease** coverage by `0.92%`.
> The diff coverage is `62.88%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6576/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6576?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6576      +/-   ##
==========================================
- Coverage   80.37%   79.45%   -0.93%     
==========================================
  Files         156      156              
  Lines       28058    28380     +322     
==========================================
- Hits        22552    22548       -4     
- Misses       5506     5832     +326     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6576?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/configuration\_reformer.py](https://codecov.io/gh/huggingface/transformers/pull/6576/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3JlZm9ybWVyLnB5) | `100.00% <ø> (ø)` | |
| [src/transformers/data/processors/glue.py](https://codecov.io/gh/huggingface/transformers/pull/6576/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvZ2x1ZS5weQ==) | `48.91% <0.00%> (-0.18%)` | :arrow_down: |
| [src/transformers/generation\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6576/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3V0aWxzLnB5) | `96.94% <ø> (ø)` | |
| [src/transformers/hf\_argparser.py](https://codecov.io/gh/huggingface/transformers/pull/6576/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9oZl9hcmdwYXJzZXIucHk=) | `67.74% <0.00%> (-1.49%)` | :arrow_down: |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6576/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `78.73% <ø> (ø)` | |
| [src/transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/6576/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0LnB5) | `88.42% <ø> (+0.16%)` | :arrow_up: |
| [src/transformers/modeling\_mbart.py](https://codecov.io/gh/huggingface/transformers/pull/6576/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19tYmFydC5weQ==) | `100.00% <ø> (ø)` | |
| [src/transformers/optimization.py](https://codecov.io/gh/huggingface/transformers/pull/6576/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb24ucHk=) | `95.55% <ø> (ø)` | |
| [src/transformers/tokenization\_utils\_base.py](https://codecov.io/gh/huggingface/transformers/pull/6576/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHNfYmFzZS5weQ==) | `93.76% <ø> (ø)` | |
| [src/transformers/trainer\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/6576/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyX3RmLnB5) | `12.25% <0.00%> (ø)` | |
| ... and [43 more](https://codecov.io/gh/huggingface/transformers/pull/6576/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6576?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6576?src=pr&el=footer). Last update [d329c9b...44e9543](https://codecov.io/gh/huggingface/transformers/pull/6576?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Neat implem! I was initially wondering whether we should just have a `HyperTrainer` whose only job would be to spawn multiple instances of Trainers, given that the adherence between hp search and everything else in this class is rather small.

Thoughts? Others feel free to chime in 

> I was initially wondering whether we should just have a `HyperTrainer` whose only job would be to spawn multiple instances of Trainers

We can also take that road, my reasoning was that since the `HyperTrainer` would need to take all the same arguments as `Trainer` and then some, it made sense to just add it to the class. I think it's easier for a user to just have one class as an access point.