add dataset split and config to model-index in TrainingSummary.from_trainer
Author: loicmagne
Pull request: yes

# What does this PR do?

Add dataset split and config to model-index in TrainingSummary when generated with from_trainer on a HF Hub dataset

example: https://huggingface.co/loicmagne/pr_dataset_metadata/blob/main/README.md 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@lewtun 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for working on this @loicmagne 🔥 !

For the failing CI, I think you first need to run:

```
make style && make quality
```

to format the code.

For the failing unit tests, it seems there's a problem with:

```
FAILED examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_speech_recognition_seq2seq
FAILED examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_swag
```

In both cases, the source of the error is a missing key in the dataset tag logic:

```
self = TrainingSummary(model_name='tmpbcyb71hu', language='en', license='apache-2.0', tags=['generated_from_trainer'], finetu...epsilon=1e-08', 'lr_scheduler_type': 'linear', 'lr_scheduler_warmup_steps': 2, 'training_steps': 20}, source='trainer')
metric_mapping = {'accuracy': 'Accuracy'}

    def create_model_index(self, metric_mapping):
        model_index = {"name": self.model_name}
    
        # Dataset mapping tag -> name
        dataset_names = _listify(self.dataset)
        dataset_tags = _listify(self.dataset_tags)
        dataset_args = _listify(self.dataset_args)
        dataset_metadata = _listify(self.dataset_metadata)
        if len(dataset_args) < len(dataset_tags):
            dataset_args = dataset_args + [None] * (len(dataset_tags) - len(dataset_args))
        dataset_mapping = {tag: name for tag, name in zip(dataset_tags, dataset_names)}
        dataset_arg_mapping = {tag: arg for tag, arg in zip(dataset_tags, dataset_args)}
        dataset_metadata_mapping = {tag: metadata for tag, metadata in zip(dataset_tags, dataset_metadata)}
    
        task_mapping = {
            task: TASK_TAG_TO_NAME_MAPPING[task] for task in _listify(self.tasks) if task in TASK_TAG_TO_NAME_MAPPING
        }
    
        model_index["results"] = []
    
        if len(task_mapping) == 0 and len(dataset_mapping) == 0:
            return [model_index]
        if len(task_mapping) == 0:
            task_mapping = {None: None}
        if len(dataset_mapping) == 0:
            dataset_mapping = {None: None}
    
        # One entry per dataset and per task
        all_possibilities = [(task_tag, ds_tag) for task_tag in task_mapping for ds_tag in dataset_mapping]
        for task_tag, ds_tag in all_possibilities:
            result = {}
            if task_tag is not None:
                result["task"] = {"name": task_mapping[task_tag], "type": task_tag}
    
            if ds_tag is not None:
>               metadata = dataset_metadata_mapping[ds_tag] or {}
E               KeyError: 'swag'
```

I suggest trying to reproduce the error locally with

```
$ pip install -r examples/pytorch/_tests_requirements.txt  # only needed the first time
$ python -m pytest -n auto --dist=loadfile -s -v ./examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_swag
```

@loicmagne I see the two same examples test are still failing:

```
FAILED examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_speech_recognition_seq2seq
FAILED examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_swag
```

Could you see if you can reproduce this locally and fix them if needed?

Fo reference, here's the stack trace coming from the speech test:

```
self = TrainingSummary(model_name='tmp0e49tqpd', language=None, license=None, tags=['generated_from_trainer'], finetuned_from...Adam with betas=(0.9,0.999) and epsilon=1e-08', 'lr_scheduler_type': 'linear', 'training_steps': 10}, source='trainer')
metric_mapping = {}

    def create_model_index(self, metric_mapping):
        model_index = {"name": self.model_name}
    
        # Dataset mapping tag -> name
        dataset_names = _listify(self.dataset)
        dataset_tags = _listify(self.dataset_tags)
        dataset_args = _listify(self.dataset_args)
        dataset_metadata = _listify(self.dataset_metadata)
        if len(dataset_args) < len(dataset_tags):
            dataset_args = dataset_args + [None] * (len(dataset_tags) - len(dataset_args))
        dataset_mapping = {tag: name for tag, name in zip(dataset_tags, dataset_names)}
        dataset_arg_mapping = {tag: arg for tag, arg in zip(dataset_tags, dataset_args)}
        dataset_metadata_mapping = {tag: metadata for tag, metadata in zip(dataset_tags, dataset_metadata)}
    
        task_mapping = {
            task: TASK_TAG_TO_NAME_MAPPING[task] for task in _listify(self.tasks) if task in TASK_TAG_TO_NAME_MAPPING
        }
    
        model_index["results"] = []
    
        if len(task_mapping) == 0 and len(dataset_mapping) == 0:
            return [model_index]
        if len(task_mapping) == 0:
            task_mapping = {None: None}
        if len(dataset_mapping) == 0:
            dataset_mapping = {None: None}
    
        # One entry per dataset and per task
        all_possibilities = [(task_tag, ds_tag) for task_tag in task_mapping for ds_tag in dataset_mapping]
        for task_tag, ds_tag in all_possibilities:
            result = {}
            if task_tag is not None:
                result["task"] = {"name": task_mapping[task_tag], "type": task_tag}
    
            if ds_tag is not None:
>               metadata = dataset_metadata_mapping[ds_tag] or {}
E               KeyError: 'hf-internal-testing/librispeech_asr_dummy'
```

To make sure all tests are now green, could you rebase on main? The examples test takes forever because of a new release of a dependency, we've pinned it on main.

CI is now green, so merging this!