KeyError: 'eval_f1'  when hyperparameter tuning distilroberta with raytune and population based training
Author: aelb66
Pull request: no

## Who can help:
ray/raytune: @richardliaw, @amogkam
trainer: @sgugger

## Information
I'm trying to hyperparameter tune DistilRoberta with RayTune using PBT and HuggingFace Trainer API. Im using Google Colab with 1 GPU to tune the model.

## Error Message 
All my trials have the same error, this is just the error for the 3rd trial

```
`(_objective pid=42097) 
 91%|█████████ | 50/55 [00:05<00:00,  8.73it/s]
(_objective pid=42097) 
 93%|█████████▎| 51/55 [00:05<00:00,  8.72it/s]
(_objective pid=42097) 
 95%|█████████▍| 52/55 [00:05<00:00,  8.72it/s]
(_objective pid=42097) 
 96%|█████████▋| 53/55 [00:05<00:00,  8.73it/s]
(_objective pid=42097) 
 98%|█████████▊| 54/55 [00:06<00:00,  8.73it/s]
                                                  
 25%|██▌       | 438/1752 [02:43<07:14,  3.02it/s]
100%|██████████| 55/55 [00:06<00:00,  8.73it/s]
                                               
2023-02-16 05:38:41,958	ERROR trial_runner.py:1088 -- Trial _objective_f0650_00002: Error processing event.

ray.exceptions.RayTaskError(KeyError): ray::ImplicitFunc.train() (pid=42097, ip=172.28.0.12, repr=_objective)
  File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable/trainable.py", line 367, in train
    raise skipped from exception_cause(skipped)
  File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable/function_trainable.py", line 335, in entrypoint
    return self._trainable_func(
  File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable/function_trainable.py", line 652, in _trainable_func
    output = fn()
  File "/usr/local/lib/python3.8/dist-packages/transformers/integrations.py", line 332, in dynamic_modules_import_trainable
    return trainable(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable/util.py", line 386, in inner
    return trainable(config, **fn_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/integrations.py", line 233, in _objective
    local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 1543, in train
    return inner_training_loop(
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 1883, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2132, in _maybe_log_save_evaluate
    self._report_to_hp_search(trial, self.state.global_step, metrics)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 1229, in _report_to_hp_search
    self.objective = self.compute_objective(metrics.copy())
  File "<ipython-input-27-3decc80fdc30>", line 3, in my_objective
KeyError: 'eval_f1'

(_objective pid=42097)                                                      precision    recall  f1-score   support
(_objective pid=42097) 
(_objective pid=42097)                                                   0      0.812     0.745     0.777       145
(_objective pid=42097)                                                   1      0.650     0.988     0.784       173
(_objective pid=42097)                                                   2      0.614     0.906     0.732       128
(_objective pid=42097)                                                   3      0.850     0.936     0.891       109
(_objective pid=42097)                                                   4      0.593     0.273     0.374       187
(_objective pid=42097)                                                   5      0.818     0.115     0.202        78
(_objective pid=42097)                                                   6      0.584     0.653     0.617       239
(_objective pid=42097)                                                   7      0.606     0.434     0.506        99
(_objective pid=42097)                                                   8      0.596     0.738     0.659       408
(_objective pid=42097)                                                   9      0.564     0.175     0.267       126
(_objective pid=42097)                                                  10     0.731     0.831     0.778        59
(_objective pid=42097) 
(_objective pid=42097)                                            accuracy                          0.644      1751
(_objective pid=42097)                                           macro avg      0.674     0.618     0.599      1751
(_objective pid=42097)                                        weighted avg      0.647     0.644     0.612      1751
(_objective pid=42097) 
(_objective pid=42097) {'eval_loss': 1.00838041305542, 'eval_macro_f1': 0.598748987658475, 'eval_macro_precision': 0.6744155267629177, 'eval_macro_recall': 0.6175753130931809, 'eval_balanced_accuracy': 0.6175753130931809, 'eval_runtime': 6.2857, 'eval_samples_per_second': 278.568, 'eval_steps_per_second': 8.75, 'epoch': 1.0}
(pid=43454) 2023-02-16 05:38:44.739485: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
(pid=43454) 2023-02-16 05:38:44.739641: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
(pid=43454) 2023-02-16 05:38:44.739655: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

== Status ==
Current time: 2023-02-16 05:38:47 (running for 00:08:54.32)
Memory usage on this node: 13.8/83.5 GiB 
PopulationBasedTraining: 0 checkpoints, 0 perturbs
Resources requested: 10.0/12 CPUs, 1.0/1 GPUs, 0.0/49.72 GiB heap, 0.0/24.86 GiB objects
Result logdir: /content/ray_results/tune_transformer_pbt
Number of trials: 20/50 (3 ERROR, 16 PENDING, 1 RUNNING)
+------------------------+----------+-------------------+-----------------+--------------------+----------------+
| Trial name             | status   | loc               |   learning_rate |   num_train_epochs |   weight_decay |
|------------------------+----------+-------------------+-----------------+--------------------+----------------|
| _objective_f0650_00003 | RUNNING  | 172.28.0.12:43454 |     4.87964e-05 |                  4 |     0.0102922  |
| _objective_f0650_00004 | PENDING  |                   |     1.00312e-05 |                  5 |     0.469276   |
| _objective_f0650_00005 | PENDING  |                   |     2.21697e-05 |                  5 |     0.0917023  |
| _objective_f0650_00006 | PENDING  |                   |     2.16492e-05 |                  6 |     0.215973   |
| _objective_f0650_00007 | PENDING  |                   |     1.18666e-05 |                  4 |     0.19993    |
| _objective_f0650_00008 | PENDING  |                   |     2.82428e-05 |                  5 |     0.183181   |
| _objective_f0650_00009 | PENDING  |                   |     4.93292e-05 |                  4 |     0.191231   |
| _objective_f0650_00010 | PENDING  |                   |     3.43018e-05 |                  2 |     0.0232252  |
| _objective_f0650_00011 | PENDING  |                   |     1.05306e-05 |                  6 |     0.22525    |
| _objective_f0650_00012 | PENDING  |                   |     4.23359e-05 |                  2 |     0.482816   |
| _objective_f0650_00013 | PENDING  |                   |     1.92358e-05 |                  2 |     0.00798313 |
| _objective_f0650_00014 | PENDING  |                   |     1.48815e-05 |                  5 |     0.220076   |
| _objective_f0650_00015 | PENDING  |                   |     1.69346e-05 |                  6 |     0.416597   |
| _objective_f0650_00016 | PENDING  |                   |     3.65009e-05 |                  2 |     0.12939    |
| _objective_f0650_00017 | PENDING  |                   |     1.83177e-05 |                  3 |     0.212578   |
| _objective_f0650_00018 | PENDING  |                   |     4.87834e-05 |                  5 |     0.0924272  |
| _objective_f0650_00019 | PENDING  |                   |     2.5806e-05  |                  3 |     0.224877   |
| _objective_f0650_00000 | ERROR    | 172.28.0.12:39382 |     3.92798e-05 |                  5 |     0.475357   |
| _objective_f0650_00001 | ERROR    | 172.28.0.12:40740 |     2.78333e-05 |                  6 |     0.298425   |
| _objective_f0650_00002 | ERROR    | 172.28.0.12:42097 |     2.33483e-05 |                  4 |     0.229624   |
+------------------------+----------+-------------------+-----------------+--------------------+----------------+
Number of errored trials: 3
+------------------------+--------------+---------------------------------------------------------------------------------------------------------------------+
| Trial name             |   # failures | error file                                                                                                          |
|------------------------+--------------+---------------------------------------------------------------------------------------------------------------------|
| _objective_f0650_00000 |            1 | /content/ray_results/tune_transformer_pbt/_objective_f0650_00000_0_num_train_epochs=5_2023-02-16_05-29-52/error.txt |
| _objective_f0650_00001 |            1 | /content/ray_results/tune_transformer_pbt/_objective_f0650_00001_1_num_train_epochs=6_2023-02-16_05-32-48/error.txt |
| _objective_f0650_00002 |            1 | /content/ray_results/tune_transformer_pbt/_objective_f0650_00002_2_num_train_epochs=4_2023-02-16_05-35-45/error.txt |
+------------------------+--------------+---------------------------------------------------------------------------------------------------------------------+ 
```

## Hyperparameter tuning code

```
def compute_metrics(p): 
...
  return {
      'macro_f1' : macro_f1, 
  }


#hyperparameter tuning configs

def my_objective(metrics):
    return metrics["eval_f1"]

training_args = TrainingArguments(
...
        metric_for_best_model="eval_f1",
    )

def model_init():
     return AutoModelForSequenceClassification.from_pretrained(model_checkpoint,num_labels=11,)

trainer = Trainer(
...
        compute_metrics=compute_metrics,
    )

tune_config = {
        "per_device_train_batch_size": 32,
    }

scheduler = PopulationBasedTraining(
...
        metric="eval_f1",
...

#hyperparameter search
best_run = trainer.hyperparameter_search(
        hp_space=lambda _: tune_config,
        compute_objective = my_objective,
...
    )

```


Any help would be greatly appreciated!

Comments:
Please use the [forums](https://discuss.huggingface.co/) to help debug your code as we keep issues for bugs and feature requests only. Here the metric you are using with `metric_for_best_model="eval_f1"` does not exist as your `compute_metrics` function only returns the following keys:
```py
{
      'macro_f1' : macro_f1, 
      'macro_precision': macro_precision,
      'macro_recall': macro_recall,
      'balanced_accuracy': acc
  }
```
`eval_macro_f1` would work better.


Thank you and sorry about that!