Train with multiple eval datasets raises an Exception
Author: eyalmazuz
Pull request: no

### System Info

Transformers: 4.25.1
Python: 3.10.8
OS: Manajro-KDE

### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Create the following compute metrics method:
```def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    accuracy_score = accuracy.compute(predictions=predictions, references=labels)
    auc_score = auc.compute(prediction_scores=logits[:, 1], references=labels)
    f1_score = f1.compute(predictions=predictions, references=labels)
    aupr = average_precision_score(y_score=logits[:, 1], y_true=labels)
    return {**f1_score , **{'PR-AUC': aupr}, **accuracy_score, **auc_score}
```
2. Create a trainer object with multiple eval datasets:
```
training_args = TrainingArguments(
    output_dir="./results",
    learning_rate=2e-5,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    weight_decay=0.01,
    evaluation_strategy=IntervalStrategy.STEPS,
    report_to='wandb',
    run_name='ChemBERTa Test',
    log_level='critical',
    logging_steps=1, 
)
```
```trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset={'Validation': dataset["validation"], 'Test': dataset["test"]},
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)
```
4. Run the trainer
``trainer.train()``

Result:
![image](https://user-images.githubusercontent.com/34383384/205439136-0364b8fe-3e71-488b-b1a4-acdcdd9bef3e.png)
![image](https://user-images.githubusercontent.com/34383384/205439150-34a5fab9-9775-4771-8a6e-1d33ebed36fd.png)



the problem arises in ``transformers/utils/notebook.py/``
in line 244, it raises ``KeyError: 'Validation F1'``
The reason for that is that only on the first iteration it populates the names of the columns for the table it prints in the notebook (lines 237-243).
But the trainer runs each dataset in a standalone ``self.evaluate`` run
So when the 2nd ``self.evaluate`` with the test dataset calls 
``self.callback_handler.on_evaluate(self.args, self.state, self.control, output.metrics)``
the table already has the names of the columns from the validation set, so lines 237-243 don't run again
and then on line 244, since it's the test dataset, all the metrics are called "Test <metric>" so it raises an error since it iterates over the table columns and doesn't find the validation metrics.


A workaround is removing the if statement in line 238 and having it update columns each time it's called
and changing line 244 to: ``self.inner_table.append([values[c] if c in values else 'NaN' for c in columns])``

i.e.
```
    def write_line(self, values):
        """
        Write the values in the inner table.

        Args:
            values (`Dict[str, float]`): The values to display.
        """
        if self.inner_table is None:
            self.inner_table = [list(values.keys()), list(values.values())]
        else:
            columns = self.inner_table[0]
            # if len(self.inner_table) == 1:
            #     # We give a chance to update the column names at the first iteration
            for key in values.keys():
                if key not in columns:
                    columns.append(key)
            self.inner_table[0] = columns
            print(f'{columns=}, {values=}')
            self.inner_table.append([values[c] if c in values else 'NaN' for c in columns])
```
(the print is only for debugging)

but then it creates a weird table where the rows are duplicated at the logging step
and half of the columns in the first row are NaN
and the other half is NaN in the 2nd row.

![image](https://user-images.githubusercontent.com/34383384/205439041-587335cb-0004-4b8e-8e7f-2f1d76c549be.png)

edit: Another bug is on ``NotebookProgressCallback.on_evaluate`` lines 342-345
```
for k, v in metrics.items():
                if k == f"{metric_key_prefix}_loss":
                    values["Validation Loss"] = v
```

This forces the output to consistently be named validation loss, even if I use the test dataset; this means that in the table created I don't get 2 different losses printed, cause the test loss overrides the validation loss


### Expected behavior

A single row containing both the validation and test metrics

Comments:
A workaround (or solution) for the problem that I found is as follows:

1. Add an optional parameter to evaluate the method in trainer called ``multiple`` and default it to False
2. change the line of code logging happens from:
```
self.control = self.callback_handler.on_evaluate(self.args, self.state, self.control, output.metrics)
```
to:
```
if not multiple:
    self.control = self.callback_handler.on_evaluate(self.args, self.state, self.control, output.metrics)
```
3. on ``_maybe_log_save_evaluate`` change the following block from:
```
        if self.control.should_evaluate:
            if isinstance(self.eval_dataset, dict):
                for eval_dataset_name, eval_dataset in self.eval_dataset.items():
                    metrics = self.evaluate(
                        eval_dataset=eval_dataset,
                        ignore_keys=ignore_keys_for_eval,
                        metric_key_prefix=f"eval_{eval_dataset_name}",
                    )
            else:
                metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
            self._report_to_hp_search(trial, self.state.global_step, metrics)

        if self.control.should_save:
            self._save_checkpoint(model, trial, metrics=metrics)
            self.control = self.callback_handler.on_save(self.args, self.state, self.control)
```
to
```
if self.control.should_evaluate:
            if isinstance(self.eval_dataset, dict):
                all_metrics = {}
                for eval_dataset_name, eval_dataset in self.eval_dataset.items():
                    metrics = self.evaluate(
                        eval_dataset=eval_dataset,
                        ignore_keys=ignore_keys_for_eval,
                        metric_key_prefix=f"eval_{eval_dataset_name}", multiple=True,
                    )
                    all_metrics = {**all_metrics, **metrics}
                self.control = self.callback_handler.on_evaluate(self.args, self.state, self.control, all_metrics)
                metrics = all_metrics
            else:
                metrics = self.evaluate(ignore_keys=ignore_keys_for_eval, multiple=False)
            self._report_to_hp_search(trial, self.state.global_step, metrics)

        if self.control.should_save:
            self._save_checkpoint(model, trial, metrics=metrics)
            self.control = self.callback_handler.on_save(self.args, self.state, self.control)
```

Now I get the desired outcome.

![image](https://user-images.githubusercontent.com/34383384/205440003-2ce93cf9-bd37-4d5a-8803-ed896390ce6f.png)


edit: this solution breaks the tqdm progress bar

Thanks for the report. The Trainer does not support multiple evaluation datasets when in a notebook indeed.
As a workaround, you can also disable the notebook progress bars by doing `trainer.remove_callback(NotebookProgressCallback)` to avoid having the issue.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

If I were to disable the notebook progress bars by doing` trainer.remove_callback(NotebookProgressCallback)`, then how do I view/retrieve the evaluation results on the multiple datasets (without using print statements or logging)?

I think the multiple dataset notebook support will be added here: #25796