Supporting multiple evaluation datasets in `Trainer` and `Seq2seqTrainer`
Author: alexcoca
Pull request: no
Labels: name: WIP

# 🚀 Feature request

Support for evaluating on multiple validation datasets when using the `Trainer` class.

## Motivation

This is a common use case in research. Imagine that you train your model on some data, and you have validation data coming from two distinct distributions. You want to compute two sets of metrics - one for the validation dataset with the same distribution as the training data and one for the validation dataset with known  distribution.

## Your contribution

- Happy to submit an example with my own code (assuming the research makes sense) so that others see how this can be achieved in practice. 

- Could update relevant posts on the huggingface forum so that other users requiring this feature can see how it can be done.

## Things I have tried

Inspired by this post [here](https://discuss.huggingface.co/t/evaluating-your-model-on-more-than-one-dataset/1544) and @sgugger's solution, I set out to see if implementing the `on_evaluate` callback is possible, but I can't figure out how to get the other validation datasets to it - the callaback can only access the trainer init arguments/state but none of these objects can be passed additional data loaders. 

How can this be approached instead? My current solution is not clean, but may work:

1. Use `setattr` to add an attribute to the trainer after init, call it `additional_eval_datasets`
2. Override the `_maybe_log_save_evaluate` method as follows:
         - Call the `Trainer` superclass method first to do what the trainer would normally do
         - loop through the additional datasets, calling the `Trainer.evaluate` for each dataset with appropriate inputs

This is a bit hacky as I should not be overriding a private method, but override `evaluate` instead. However, the implementation would be more concise in this way. Please feedback of any undesirable side effects that this may lead to - I did read through the `Trainer` source code and did not spot any pitfalls!

Of course, this approach can be adapted in order to support evaluation on multiple datasets natively in the `Trainer`.   



Comments:
We could support several evaluation datasets inside the `Trainer` natively. I think the easiest would be to:
- accept a list of datasets for the `eval_dataset` at init
- have a new boolean `TrainingArguments` named `multiple_eval_dataset` that would tell the `Trainer` that it has several evaluation datasets (since it won't be able to make the difference between one or several datasets: it could very well receive a list for a regular evaluation dataset).

Would you like to work on a PR for this?

Hi @sgugger,

Yes, that would be a nice solution. Happy to try and prototype this in my own time (weekends, overwhelmed PhD student here) and see if we can put together something for one of the future releases. 

Just to make sure I understand the design principle: first we make the above changes and then change the logic inside `_maybe_log_save_evaluate` to deal with multiple datasets? This requires some assumptions (e.g., report the metrics to hp search from evaluation on dataset at index `0` or some user specified index or average over specified indices). These would also have to be defined as additional arguments to `TrainingArguments` so that we know what to report to the hyperparameter search.

The alternative is that we just make the changes you mentioned, which would allow the user to write the  `on_evaluation` callback - an example of how that callback looks like could be in the docs.  

Which is the preferred option?

The idea would be to loop other the datasets when calling the `evaluate` function in `_maybe_log_save_evaluate` all natively inside the `Trainer`.

We can also have another `TrainingArguments` for the names of the datasets, that's a good idea, which would defaults to just 0, 1, etc...


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

not stale. I'll pick this up soon enough!


I'll put the `WIP` label so that it is not closed!

Hi! I also need this feature and I have a hacky implementation to make it work (based on @sgugger 's suggestions).
How is the status on your side @alexcoca? Just to know if it's worth polishing it up on my side and make a PR.

@bmichele , really nice! I've been busy with conference deadlines so progress has been slow. How about you open a PR and I can also help make some contributions/suggestions so we get the job done?


Is there an update on this?:)